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ABSTRACT 

A Study conducted of Title I evaluation practices in 
urban districts which comprised the membership of the Council of the 
Great city Schools and to identify special problems rela^d to Title 
I evaluation, as well as make recommendations for improving 
evaluation practices, is presented. The report contains the following 
sections: I. Evaluation of Title I Projects; Problems and Needs; II. 
Summary of On->Site Visits: Procedure and Results; III. Summary of 
Questionnaire Results; IV. Summary of Conference Results; and 
Appendixes. The appendixes are: Combined Questionnaire for Research 
Directors and Title I coordinators; Lake Placid Conference. (DB) 
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In April, 1970, the Council of the Great City Schools was 
awarded a contract by the Department of Health, Education and 
Welfare to survey member school districts * relative to certain 
aspects of Title I of the Elementary and Secondary Education Act 
of 1965. Two of the more specific purposes of the study were to 
study Title I evaluation practices in the urban districts which 
comprised Council membership and, from the perspective of local 
district personnel, to identify special problems related to 
Title I evaluation and make recommendations for improving evalua- 
tion practices. 



*The authors are indebtbd to the following persons who were 
staff members of the Council of the Great City Schools at the 
time of the study and who assisted In its conduct: Marvin Dawson, 

William T. Denton, Lora Liss, Gerald Colendihe, and Edward N. 

Whitney. Member districts at the time of the study were: Atlanta, 

Baltimore, Boston, Buffalo, Chicago, Cleveland, Dallas, Denver, 
Detroit, Los Angeles, Memphis, Milwaukee, Minneapolis, New York City, 
Philadelphia, Pittsburgh, St. Louis, San Diego, San Francisco, and 
Washington, D. C. 
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The authors were Involved in the study respectively as 
Director of Research for the Council, consultant, and member 
of the Council’s Research Advisory Committee. 

The project began May 1, 1970, and it ended with delivery 
of a final report on July 29, 1970. It was conducted in three 
major task groups, as follows: 

1. A telephone survey in which the federal program 
director from each of the twenty member districts 
was asked to identify and forward reports on the 
two **best*' Title I projects of the 1968-1969 school 
year. The federal director used his own criteria for 
"goo4*“ The research director in each district was 
asked to identify and forward reports on the two 
"best" Title I evaluations. The evaluation did not 
have to be for the projects chosen by the federal 
program director. 

2. Site visits to four meml r districts. During these 
visits, information was gathered from personnel in- 
volved in Title I projects. Information from the 
first twc* visits was used to develop two questionn- 
aires to be administered to the research directors and 
the Title I program directors. The final two site 
visits were used to validate the questionnaires and 
gather additional Information. 

3. A conference held at Lake Placid, New York, early in 

* June. Prrticlpants at this conference Included re- 

search and evaluation personnel, federal program 
directors, other administrators, and teachers from 
the member school districts. Each participant was in- 
volved In some key way In Title I programs. This 
unique mixture of personnel representing a number of 
different viewpoints was used to gather additional 
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Information relative to Title I projects and Title 1 
evaluations* 

• • • 

The information generated through the above activities served 
the input for this report. 
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I, Evaluation of Title I Projects: 

Problems. and Needs 

« 

Evaluation of Title I projects has been required since the very beginning 
of the program. The original guidelines stated that every project had to be 
evaluated » and considerable resources, both financial and human, have gone in- 
to evaluation efforts. It is an understatement to say that these efforts have 
not met with complete success, either at the local, state, or national levels. 

At the most general level, the problem has stemmed from lack of adequate 
methodology for the task at hand. The guidelines stated that a wide range of 
research-like questions were to be answered about Title I projects, but they 
set a different context from that traditionally faced by researchers. Matters 
of research — or “evaluation” — were not to Influence the planning of pro- 
grams. To draw from the language actually used, the tail was not to wag the 
dog. This meant that many traditional controls were not useable. 

Evaluation was attempted, but the lack of an adequate theoretical and 
methodological base doomed the early attempts to failure. Egon Cuba, in the 
March 1969 issue of Educational Researcher , voiced the feeling of frustration 
of a good part of the educational research community with his statement that 
“evaluative inquiry does little more for the researcher than to give him an 
opportunity to disgrace himself while not really serving his purposes.”^ 

As we shall see, this frustrating lack of success has had a positive 
long-range effect by making clear the need for new methodologies and by 
fostering related developmental efforts. A number of very capable people have 
recently been engaged in building theories of evaluation and in developing 
appropriate methodologies. Recent books, such as Educational Evaluation and 
Decision Making , by Stuff lebeam and the Phi Delta Kappa study group; Dis- 
crepency Evaluation , by Prevus; the Education of Instruction , by Wittrock and 
Wiley; and individual titles in the AERA monograph series, attest to major ac- 
complishments in this area. A purpose of the Great Cities project was to aid 
this effort by identifying and discussing evaluation problems as they are 
seen by people in the field. Evaluation by its nature is field-oriented, and 
any theory or set of methodologies which does not account for the concerns of 
people who work in the field is certain to be incomplete and its products of 
limited usefulness. 



^Guba, Egon, “Significant Differences, “Educational Researcher . 
XX:3, March 1969, p. 5. 



« Five general areas of concern ware identified in that part of the project 

* addressed to evaluation: planning and funding, design and implementation, im* 

the decision process, personnel, and state and federal relationships* 

A. Planning and Funding - 

The feeling was strong among people who made inputs to the project that 
many difficulties in evaluation of Title I projects stem from inadequate plan- 
ning before projects begin and, in a related matter, from inability to con- 
sider long-range approaches because of funding uncertainties* 

Planning for evaluation has been weak because, in apparently a large 
number of instances, no person with evaluation expertise has been included on 
the planning team* The suggestion that the tail not wag the dog has been taken 
quite literally, in other words, and evaluation has often^ not been considered 
until projects were actually being implemented* This has meant that there were 
often no evaluation design considerations whatever, even those which would have 
little effect on the operation of the program* It has also meant a late start 
in detailing information needs, in locating and building suitable Instruments, 
and in designing methods for information collection* The evaluator has there- 
fore found himself under a severe time constraint and has sometimes been beyond 
the point when certain information should have been collected* I*ack of develop 
ment time has forced a frankly shoddy job in Instrument development in far too 
many instruments* Clearly, evaluation needs should be identified as early as 
possible, and an evaluator should be a part of the planning team* 

Another planning difficulty has revolved around poor statement of* 
goals and objectives* Goals and objectives have often been "fuzzy** and not 
stated in precise enough terms to lend themselves to measurement* At a some- 
what different level, they have often been irrelevant to the real purpose of 
the project* A tendendy to select ready-made objectives from those suggested 
in state and federal guidelines was evident. Thus, a project's stated object- 
ives may appear to fit state and federal* priorities, but they may have little 
relationship to what the project is really intended to accomplish* Product 
evaluation built around such irrelevant objectives is, of course, almost cer- 
tain to indicate that a project is not successful, whatever its real effects 
may have been* Objectives may also adhere too strictly to traditional goals, 
and yet be clearly stated* A great deal more attention is needed to deriving, 
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that is, initially stating and systematically revising, meaningful objectives. 

Another problem is failure to adequately define project procedures. This 
may be related to poorly stated objectives, as well as to numerous other causes. 
An outcome is that a project is in reality likely to be no project; teachers 
will not understand what is expected of them and will just do what they other- 
wise would have done. .Thus, the treatment variable will have no operational 
meaning. This problem also affects the chances for good **process.** As stated 
below, there was strong feeling among all project participants that process 
evaluation techniques need to be improved and used a great deal more. Process 
evaluation is, of course, tied directly to program procedures and their im- 
plementation and efficacy. If program procedures are poorly defined, meaningful 
process evaluation is unlikely. 

A planning problem identified by several participants was the failure to 
gear evaluation designs to individual project needs. i.n large numbers of situa- 
tions, attempts are apparently made to use the same basic evaluation design for 
every project, and the design may have involved simply securing pre- and post- 
test results from Instruments of doubtful relevancy. In reality, evaluation 
needs differ according to a number of project characteristics. According to 
the complexity of a project, for example, the amount and type of feedback 
needed for management control will vary. According to the level of uncertainty 
related to treatment effects, to inservice needs and procedures, and to other 
project components, information needs will differ. Other things equal, the re- 
sources used for evaluation of a project should relate to the amount of re- 
sources allocated to the project itself, and even this simple rule-of-thum|? is 
often not followed. 

Participants in the study had a good deal to say about funding of evalua- 
tions. Ihe lack of clear direction from state and federal levels has created 
uncertainty, and the proportion of Title I funds allocated for evaluation has 
varied greatly just in the Great Cities. The quality of the evaluation effort 
is certain to vary drastically in this situation. The feeling is that, as a 
general rule, at least five percent of project resources should go into evalua- 
tion. Until guidelines give clearer direction, however, this rule will not be 
followed. The competition for funds is keen, particularly in the large cities 
which are experiencing financial difficulties anyway, and, coupled with the 
general feeling that evaluation has not proved very helpful in the past, the 
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temptation is strong to put as many of the resources as possible into programs 
and to do a perfunctory Job in meeting the evaluation requirement. The self- 
defeating nature of this approach » ao‘ far as any me&nlngful evaluation is con- 

earned is evident, and is recognized as such. Federal program directors, 

\ 

evaluation specialists, and others with relevant decision responsibilities beg 
for more clear direction as to the proportion of funds which should be allocated 
for evaluation. 

A related problem is the indefiniteness of funding from year to year and the 
lateness In the school year of being informed of appropriation amounts. longi- 
tudinal evaluation designs are needed for many programs and yet cannot be planned 
under current funding arrangements. Even with the year-to-year situation, actual 
amounts of appropriations are often received so late in the school year that 
adequate planning for the next year is impossible. 

s • . * 

B. Design and Implementation 

The most general problem identified in the design area is the seeming lack 
of a theoretical base for evaluation or, to state it in a different way, the 
lack of an overall evaluation strategy. Because of this situation, the purpose 
of evaluation is often not clear, and planning is haphazard. 

Two more specific methodological problems are pupil mobility and assessing 
multiple project effects. Title 1 projects are, of course, cirected toward 
pupils from low- income families, and in most locations these are the pupils with 
the highest mobility rates. In a specific project, a high proportion of subjects 
may move in the course of a school year, and evaluation designs which depend on 
assessing long-range effects on individuals are practically impossible to implement. 
Another difficulty is that program effects may not have time tp operate, so that 
basing the assessment on whoever happens to be in the project at measurement time 
is not a solution. Help in solving this problem in a manner not excessively cost- 
ly is neeried. 

Assessing multiple project effects is another difficult problem. Many child- 
ren will be in more than one Title I project, and in any event all will have a 
number of other school-related experiences. The effects of these experiences 
interact, and assessing the effects of a single project or a specific set of 
projects is practically impossible without the ability to assign students and 
use some kind of experimental design. 
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Measurement poses a difficult problem In much of Title I evaluation* A 

strong tendency has been to use what existed, that ls^ to use the results of 

the testing program of the school district*. This has led to an overreliance 

•on standardized tests and has caused a focus which has been too narrow and too 

restrictive. Measures of effect, of process, of classroom interaction, and of 

other such variables are often needed and not available , and, even, when one 

« 

is focusing on a basic skill such as reading, the relevant behaviors may not 
be adequately covered in a standardized test* Often such effects as organl* 
zational change, teacher attitude change, and change in the relationship of 
school and community go completely unmeasured* As indicated above, problems 
of stating objectives, of inadequate preplanning, of funding, and of timing 
mitigate against construction of needed instruments* 

Another methodological problem which has been widely recognized but not 
solved is the need to include fiscal data in evaluations* "Cost-effectiveness'* 
are spoken of frequently but seldom used in Title 1 evaluations* A simple, 
understandable, and practical method for tying cost figures to other evaluative 
data is needed. 

A whole set of problems exists in what might be called the "efficiency" 
area. Often, there is apparently little effort to coordinate evaluation ef- 
forts internally and to coordinate with state and federal requirements. This 
results in excessive testing of children, and an unnecessary time burden is 
imposed on teachers and administrators* Information systems which meet all 
needs on a timely basis, but without duplication, are needed but poorly 
developed in most districts at chis time* Help In planning, developing, and 
implementing such systems is needed* 

Several study participants spoke of role conflict as a problem* Ap- 
parently the place of the evaluator in the organizational structure and the 
nature of his relationship with program managers and other decision makers is 
unclear and causes communications problems* Lines of authority are not under- 
stood, and the independence of the evaluator, along with the integrity of his 
work, is threatened. At the other extreme, evaluation often exists as an iso- 
lated operation with little connection to the rest of the district Organization 
and subsequently with little connection to the planning and management decision 
processes. Suggested organizational’ patterns for locating the evaluation 



tfunctlon are needed. 

Inability to Intplement longitudinal designs was referred to above In the 
discussion on funding. In addition to hindering a .rational developmental pro** 
cess, this Is seen to cause an over-emphasis on one-shot studies. The possi- 
bility of verification of Important findings through replication Is greatly 
needed. 

Inability to Implement longitudinal designs was referred to above In the 
discussion on funding. In addition to hindering a rational developmental pro- 
cess, this Is seen to cause an over-emphasis on one-shot studies. The possi- 
bility of verification of Important findings through replication Is greatly 
limited. 

Process evaluation Is a major sub-area of difficulty. As noted previously, 
study participants, whatever their roles, agreed that greater emphasis on 
process evaluation Is needed. Yet, they noted, guidelines do not promote 
process evaluation, and techniques for Implementing It In an economically feas- 
ible way appear not to exist. Current process evaluation techniques were seen 
as time-consuming, costly, and In need of expertise not generally available. 
Since it Is likely to Involve observation In the classroom, it appears threaten- 
ing to teachers and program administrators who do not understand its purpose. 
Further, observers In the classroom interfere with the Instructional process 
and thus may affect what Is being assessed. If the number of direct observe- 
tions Is reduced, the outcome Is likely to be an additional record-keeping 
task Imposed on teachers. Help In this area Is badly needed* First, as noted, 
workable process evaluation methodologies should be developed. Currently . 
available models are simply too costly In most Instances. Ways of Informing 
line staff about process evaluation and Its purpose Sie needed. Ways of specify 
ing the type of process evaluation needed In specific situations should be 
developed. Finally, necessary funding must be provided. 

C. Impacting the Decision Process 

The third general problem area deals with impacting the decision process 
related to Title I program planning and management. Evaluators and decision 
makers alike agree that In far too few Instances have the evaluation process 
and evaluation results had any effect on program planning and program management 
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This, of course, makes a travesty of the evaluation function; It should exist 
as a service to decision making, for it serves no intrinsic purpose at the 
public school level. 

Study participants noted that evaluation and program management are often 
separated in such a way that evaluation is isolated, with its role unclear 
either to evaluators or to program personnel. In these instances, it does not 
serve its intended purpose because of misunderstandings and, related to these, 
poor performance. That is, it often does not feed key Items of information to 
the people who really need them at the proper time. This problem clearly has 
two sides. First, evaluators need to learn to do their own Jobs better, but, 
second, decision makers need to understand the evaluation process better and 
learn to use the results it produces. 

To take the second point first, there were a number of comments to the 
effect that decision makers dc not understand evaluation well enough. They are 
not oriented to the evaluation process, and they do not understand how It can 
be useful to them. This results, participants noted, in a lack of commitment 
on the part of decision makers to make use of Information produced through 
evaluation. A number of participants suggested that teachers, administrators, 
program managers, and other decision makers need to be ’’educated’* on the ways 
evaluation can be used to improve program planning and management. 

A related point is that political considerations are often the major 
determinant in making decisions. No one suggests that politics Is not always 
going to be a component of decision making, but the point is that it should 
be kept in proper perspective and that evaluative infcnoation should also play 
its proper role. 

In the same general area, problems are caused by an over-emphasis on 
positive results. Program managers and developers believe In what they are 
doing, and they want results to support what they firmly believe is right. 
Evaluation in this case poses a threat, largely because its purpose is mis- 
understood, with l:he result that evaluators do not get the cooperation they 
need and are placed under undue pressure to show positive outcomes. Again, 
the suggested solution is education of relevant decision makers to the pur- 
poses and limitations of evaluation. 

The other side of the coin is that evaluator:. te often fail to pro- 
duce the information which is needed. They fall to Idei^tify the key decision 
makers and to interact with them, and they fail to differentiate the information 
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needs of decision makers at different levels of the planning and management pro- 

cess. The same rcpart will be given to the superintendent of schools and to 
the classroom teacher, for example. Clearly, decision makers at different levels 
need different types of information and they need different reporting formats. 

The evaluator's re ‘fjsibility is to identify relevant decision makers, determine 
the type of information needed at different levels and at different points in 
time, and determine the most useful formats for transmitting it. The feeling 
was that most evaluation units need help in designing more effective reporting 
systems. 

Some study participants suggested that decision makers at times do not use 
evaluation results because of lack of confidence in the information given to 
them. This can result from misunderstandings, from failure of evaluators and 
decision makers to interact, from poor reporting practices, and, it was noted, 
from such things as fafllure to report results in terms other than standardised 
test results. 



D. Personnel 

Study participants noted one problem which has been recognized nationally 
and which therefore should come as no surprise to readers of this report-. The 
problem is a shortage of personnel specifically trained to plan, implement, i id 
conduct evaluation activities at the public school level. Persons trained in 
traditional research methods are simply not prepared, and those with the most 
rigorous training sometimes prove least useful to an effective evaluation ^ 
process. 

As noted previously in this paper, and as recognized in the literature, 
research and evaluation serve different purposes, and while they utilize some 
of the same techniques, they appear to require a different theoretical base. 
Training programs now In existence do not seem to prepare persons for evalua* 
tion as a major area of endeavor, and this deficiency is keenly felt. Special 
evaluation training programs are the suggested remedy. 

E. State and Federal Relationships 

1 • 

The last general area of difficulty identified by study participants was 




relationships betx^fcen the local exlocation agency and state and federal agencies. 

$ 

Part of the problem relates to guidelines. Tlie feeling .was chat evaluation 

% 

guidelines are far too incomplete. Frequent changes In guidelines were cited> 

as were conflicts on some points between state and federaj. requirements. An 
* 

additional point was* that guidelines are. too restrictive to allow sufficient 
local flexibility, and that most guidelines are more suitable for administra- 
tion in smaller districts than in the large city school system. 

A great deal was said about the national evaluation surveys of the past two 
years. The relevance of some of the information collected in this survey is not 
clear, and it was generally felt that the timing of instrument administration 
has been bad. A proliferation of apparently uncoordinated survey instruments 
from the federal level was noted. A good many of the local representatives felt 
that they did not understand well enough the purpose of the national evaluation 

effort. A more specific complaint was that the criteria for selecting schools 

# 

for the national survey are not clear enough. Finally, participants wondered 
why results from the national evaluation are not fed back to the local level. 

A question was raised as to why local, state, and federal evaluation ef- 
forts are not better coordinated in some way. Local evaluations at present 
seem to have no relationship to those conducted by the higher level agencies. 
Related to this, a feeling of lack of confidence at the federal level in evalua- 
tion conducted at the local level was noted. 

A specific point was the lack of technical assistance in the evaluation 
area from the federal level. Another was the frequent change of personnel 
within the Office of Education. 



F. Positive Forces In Evaluation 

Anyone rending the immcdintoly preceding section of this report might not 
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wontler at the complaint that there is little confidence at the federal level in 

• • 

local evaluation efforts. Many px*oblems are discussed, and one might suppose that 
local evaluations of Title I have been a complete failure. This impression would 
be false, for a great deal has been" accomplished. The previous section Vi^as de- 
liberately problem centered, and difficulties were discussed as a basis for 
recommendations for actions to affect improvements. To balance the picture, 
some of the positive outcomes of local Title I evaluation efforts should be 
mentioned. 

• « 

One definite advantage is the improvement in local research and evaluation 
staffs. In most school districts, staffs have grown significantly in numbers 

and in sophistication since the introduction of Title I more than six years ago. 

• • 

Related to this is a growing level of understanding and support of evaluation 
efforts on the part of both professional personnel and community. The previous 
material notes that the understanding gap is still large, but clearly signifi- 
cant strides have been made. 

The increasing emphasis on process evaluation is seen as an important 

positive effect. Six or seven years ago, few school people had ever heard of 

process evaluation; now they recognize the term and recognize its importance, 

and this emphasis almost certainly means that it will receive increased attention. 

Important strides have been made in the measurement area because of Title I 

evaluation efforts. It is now generally recognized that a wide range of program 

effects must be considered and that better ways of measuring such things as 

affective response, pupil- teacher interaction, and institutional change must be 

developed. A related area is the Increased use of behavioral objectives In 

planning projects and in outlining measurement needs. 

• *- 

A very important outcome has been recognition at the national level of 
theoretical and muthodoloj^iciil v/eak^o.ss^^. in evnluutlon. Tills has isd to Im- 



portant work which is now bearing fruit and which should soon result in iwiproved 
evaluation capability. This work might have been eventually accomplished any\^ay, 
but the hard lessons of Title I undoubtedlj speeded its occurrence. Other ad- 
vances, such as increased understanding of accountability .needs and demand for 
cost-benefit figures, can be cited. ^ 



II 



SUMMARY OF ON-SITK VISITS: t^ROCRDURK AND RESULTS 



To assist in the development of questionnaires to be given 

to research directors atid Title I coordinators » and to gather 

• ^ 

other information first-hand, site visits x^ere made to tvo member 
cities. Questions were' developed frota the information gathered 
through on-site interviews. 

« 

After the questionnaires were developed and some responses 

had been tabulated, two additional member cities were visited to 

* 

help validate the questionnaires. 

The cities chosen represented four different school administra- 
tive organizational patterns. There was a distinct differentiation 
in the roles played by research and evaluation departments. 

All of the site visits were conducted in essentially the 
same manner. Permission x^as received from the state Title I 
director to conduct the interviews. In each case two representa- 
tives from the Council spent one day in each city. The site visit 
would begin x^rith a meeting usually attended by the reseairch director. 
Title I director, assistant superintendent in charge of federally 
funded programs, and the interview team. The meeting was used to 
Inform the school district representatives of the purpose of the 
interviews and the schedule to be followed. There was also a 
discussion of the general impact of Title I programs, the influence 
exerted by evaluation reports on program design and modification, and 
the problems key administrators face in making use of evaluation 
data to make decisions about Title I projects. 

Following the meeting, the interviewers v/ould separately 
coH'vuct In depth i n t e rviev/;; v<»lth personnel involved in the Title I 
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program. 



During the interview, the Interviewee was given a copy of 

« ♦ 

the questions. The Interviewer would read each question and 
record the response. The interviewer would often ask additional 
questions to clarify a point or to follow-up subtle cues related 
to problem areas in Title I projects. 

The cooperation of local personnel was gratifying in all cases. 

A. Administrative Considerations 

One of the most obvious contrasts among the sites visited 
was the organizational pattei*n and administrative practices 
applied to Title I programs. The relative success or degree of 
satisfaction with the accomplishment of Title I projects seemed 
directly related to the way the projects were administered and 
the r.elationship of the Title I administrative unit to the total 
administrative organization of the school. 

In the first two site visits, the Title I programs were 
administered under the general organizational pattern of the school 
system in the same manner as other programs funded through State 
and Federal agencies. Usually Title I funds constituted a major 
portion of the available outside resources. Under this pattern, 
each Title I project is submitted, funded , administered and evaluated 
separately. For example, a project on teaching remedial reading 
to primary students is a separate project from a pre-'kindergarten 
project. 

♦ 

There arc som< local differences In how these separate projects 

In some canes Individuals arc assigned out of a 
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are administered. 



In*other instances 



central office to manage one or more projects, 
the admlnis tr*at Ive lines of authority work thro.ugh a school 
principal or a part-time, release time teacher. Both of these 

m 

patterns has its disadvantages. In th*e first instance, the person 
from the central office has the problem of working with, through 
and around the regular school programs to which he is only a temporar 
adjunct. In the second case the principal or teacher has a multi- 
tude of duties and assignments of which the project is only a part. 

It is easy to see the difficulties one can encounter in both cases. 

It is quite easy to subvert or ignore or down play a project at any 
stage .of its offeratlon. It is quite difficult to adequately involve 
and inform all the various administrative and teaching units 
directly and tangently Involved with a project. It is particularly 
difficult when this is tried on a part-time basis by the project 
administrator. 

In the second two site visits, a different administrative 
pattern was encountered. Due to state Title I regulations, all 
Title I projects are submitted as part of a total package. The 
administrative divisions are not on Individual projects but on com- 
ponents of the total Title I package. For example, there are pro- 
visions for instructional programs, parent involvement activities, 
auxillarv services, and inServlce training. These components cut 
across all projects and the proposal submitted must Include all these 

a 

areas across all projects. 

This pattern seems to be more logical than the individual 
project pattern. At least it provides some comprehensive way of* 
looking at the total Title T effort and a way of consolidating a 







great deal of administrative effort. 

B . Evaluation 
• • 

^ • 

The problems of assessing the impact of Title I projects 
is' almost overwhelming. While there are attempts made to 
evaluate individual projects or specific learning objectives, there 
is little or no attempt made to assess the overall impact of the 
program. 

Usually the Title I projects are evaluated by personnel from 

the Research Division of the school system. The Efficiency of 

, 

this operation varies, depending on the administrative structure 
and the personalities involved. There are some things that can 
be done to change the present pattern by bringing pressure to bear 
through legislation or project monitoring. 

The major problem seems to be the time delay between project 
evaluation and feedback to project personnel. This delay can be. 
as long as six to nine months. In some instances the evaluation 
for a project ending in June may not be available until December 
or later. This delay coupled with the need to begin writing project 
proposals or renewals in the Spring means almost a one year delay 
before the final evaluation of a project gets fed into the proposal 
process. 

This time delay is partially raused by inadequate staffing: 
by inadequate data processing and analysis and by year to year 
funding policies. It would seem that some different funding patterns 
regarding evaluative procedure is most needed. In addition much 
more emphasis on process evaluation is needed. The combination of 

. lilS 
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a change in funding pat^tern and a^ greater emphasis on process 
evaluation should increase the efficiency with, which the Title 1 
reports are assessed. Although and should be some 

ft 

changes in th\i present evaluation practices within each school 
system, this still does not address jltself to the problem of 
measuring the total impact of Title I on a' school system or systems. 
This is an area which needs further exploration. There are many 
difficult and complex questions which need to be carefully explored 
before launching any massive efforts in this area. But, it is 
the question which must ultimately be answered and one for which 
little evidence has yet been gathered. • 

C. Other Impressions 

There seems to be general agreement that there are beneficial 
side effects from the Title I program. These include the increased 
awareness on the part of school administrators of the role and 
need for tightly written and well evaluated projects. In some 
instances the pattern of evaluation established for Title I pro- 
jects has carried over into other school programs. In addition 
there has been a major expansion of the staff assigned to research 
and evaluation. It is possible in some instances to use these peopl 
in evaluating other programs as well. The increased dtaff and 
facilities for evaluation has had a definite beneficial impact on 
the schools. 

Another side effect not so easily identified or measured is 
the impact on the community. In quite a few of the Title I projects 
community Invoivemont is a major part of the program. These efforts 
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carry over into other aspects of schoo.l- community relations. 

Leaders and workers identified through Title I prog>rams continue 

• • * 

their interest and Involvement even if programs are dropped or 

* 

expand their efforts into other areas of school or community 
ac.tivities. 

* ft 



III. SUMMARY OF QUESTIONNAIRE RESULT^ 



As indicated In previous sections, questionnaires were administered to 
the federal program director and the research director from each city. The 
questionnaire instruments were developed through the site visits, as previous- 
ly escplained. 

The questionnaires were designed to gather information about the ex- 
emplary projects and Title I programs in general. Questionnaires given to 
the research directors contained two areas not included in those given to the 
Title I coordinators. The areas were staff adequacy and budgetary considerations 
There were five general areas in common, that is, given to all respondents. 
(Questionnaire format and item responses are given in Appendix A.) 

This section of the report consists of a detailed discussion of question- 
naire results and is arranged in the order that items are listed in Appendix A. 
Information here was part of the basis for the interpretative report on evalua- 
tion given as Section I of this document. 



A. Successful Evaluations 



As night be expected, both the* reeeatch directors and the 
Title I coordinators agreed that thef success of .Title I evalnations 
depends heavily on project .object-lves being stated clearly and 
in behavioral terms. This emphasis on project objectives was 
supported in all the Information gained during this project. 

The research directors said that it was easier to do a good 
evaluation if the project focused on basic skills ‘and that the 
attitude of teachers and administrators was Important to the success 

of the evaluation. 

It is interesting to note that the Title 1 coordinators placed 
more importance on the availability of standardized measurement 
devices and the project being built around a strong research design 
than did the research directors. This may, however, be due to the 
different context in which the coordinators were asked to respond 
to the question. (Coordinators were asked to rate each question 
relative t.o criteria for selecting a good evaluation for a Title 
I project; whereas, research directors were asked to rate each 
question relative to the two project evaluations which were 

identified. ) 

The items listed under other factors which were Important to 
the successful evaluation are all useful. The only comment to be 
repeated was the necessity of good computer support. 

Be Evalu^ition Problems 



1 . 



Major Problems 

Both groups of respondents agreed that inadequate instruments 
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to measure project Impadt and the time lag betveen evaluation 

I 

and input requirements have been major problems, in evaluating 
Title I programsa An interesting difference occurred when 
considerably more research directors than coordinators identified 
ambiguous project objectives and evaluation perceived as a threat 
as major problems in evaluating projects. (Apparently most of the 
coordinators feel that they are writing clear project objectives, 
but there just aren’t adequate instruments to measure them.) 

About half the coordinators identified the inability to effectively 
cross-valldate comparable projects and the lack of a data bank of 
pupils’ socio-economic level as major problems in evaluations. 

Although neither group Identified as a major problem the inability 
to employ a competent staff, the research directors in a later 
question (staff adequacy) pointed up the need for additional staff 

members. 

2. Non-measured Effects 

There was close agreement between the coordinators and 
research directors in the identification of non-measured effects. 

The effect most often identified was Improvement in school-community 
involvement. This was followed by changes in parental attitude, 
changes in school administrative practices, substantial parental 
involvement, positive changes in attitude about evaluation, and 
reduction in pupil mobility. The growth of student self-esteem was 
cited as a non-measured effect by two respondents. 

3. Factors Which Have a Negative Effect on or By Their Omission 
Have Diminished the l^^valuation Effort 

The problem of pupil mobility is strongly identified by 
both the groups. About half the members in each group cited the 
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lack of attitudinal measures as having a negative effect on 
evaluation of Title I projects. 

4. Context Problems in Evaluation 

• e 

In the area of context, the problem of pupil mobility ranks 

s 

far ahead of the other choices. It is clear that both research 

directors and coordinators have difficulty handling the problem 

of pupil mobility. This is a critical problem since the schools 

in which Title I projects are usually conducted tend to have high 

« 

pupil mobility. Following pupil mobility were sampling problems, 
conflict between demands of instructional practice and demands of 
evaluation, and administrators not research oriented. The amount 
of importance each group gave additional problems dropped off 
rapidly, with little importance given to student unrest, teacher 
attitudes toward evaluation, or teacher strikes. 

5. Technical Assistance 

Most respondents said they had asked for technical assistance 
from outside sources in evaluating their Title I projects. Most of 
the. sources used had proved to be useful. The source most often 
used for technical assistance was the local university, with the 
assistance being useful in nearly every case. More than 50% of the 
coordinators identified state educational officials as being used 
for technical assistance. 

6. Year-to-year Funding 

Both the coordinators and research directors agreed that 
the year-to-year funding practice of Title I placed a hardship on 
the evaluation of Title I projects. It was felt that the practice 

ft 

reduced the quality of the evaluation report, hindered staff recruitment 
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and retention, and placed a burden on the evaluation report for 

« 

re-funding of the project. In the other comnex^ts, several of the 

respondents felt that effective planning was not possible because 

• % 

of time constraints posed by the year-to-year funding policy. 
(Possible solutions to this problem might be earlier funding 
and/or funding for a longer period of time.) 

7. School-wide Effects 

. The Title I coordinators cited several school-wide effects 

of Title I projects which were not stated as objectives. They 
said Title I has become the broad catalytic agent for broad 
institutional change, has fostered school/community Interaction, 
and has been responsible for broadening the awareness of extra- 
school responsibilities. 

C. Successful Title I Projects 

To help ascertain the reasons for the success of the Title 
I projects which were identified by the federal program directors, 
a series of seventeen questions were developed. The research 
director and Title I coordinator responded to the questions 
relative to the exemplary projects Identified for their system. 

It was generally agreed that a successful project would have 
a planning committee of 15 or fewer members composed of represen- 
tatives from the community, parents, teachers, administrators and 
evaluators. 

The project would be Innovative In nature and focused on basic 

* 

skills. The project objectives would be clear and stated In 
behavioral terms. The project would likely be directed at younger 



students . 
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Project leadership was rated as very Important throughout the 
project. The leader would preferably devote full-time to the 
project, lead In a democratic fashion and be well experienced with 
similar projects. It is very important that he have clear lines of 
authority. 

An experienced staff, teachers with at least 5 or 6 years 
of experience, would be chosen for, the project. They would be 
able to devote full-time to Title I projects. The staff would 

participate in an extensive in-service training program which 

• • • • 

would be carried on throughout the project. The in-service 
training program would be conducted during school hours and be 
taught by school personnel with the aid of outside consultants. 

The staff would be supported by personnel from the community who 
were experienced in what the project was trying to do. The project 
would be likely to foster more community involvement which would 
contribute to' its success. 

There would be commercially-available materials and hardwa^re 
to be used in the project. 

Hopefully, process evaluation will be used to support the 
project. (Although coordinators feel there is more process evaluation 
being done in current projects than do the research directors* it 
was generally expressed at the conference that more process evalua- 
tion would be done in future projects.) Process evaluation should 
be Important throughout the project, and should be conducted by 
both the project managers and the research and evaluation division 
of the school district. 
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D* Project Changes 

There was general agreement* between the research directors 
.and Title I coordinators that projects have ctianged. In most cases, 
the changes were seen as. gradual rather than dramatic, although 
some projects have changed significantly. Both groups said that 

m 

the nature of the changes had been toward more emphasis on basic 
skills and early childhood education. The Title I coordinators 
felt that changes had also been toward more emphasis on innovative 
programs, more emphasis on child growth and development, and more 
continuity programs for target pupils. 

Most project management decisions are made by the project 
manager, with the rest of the decisions being made primarily by 
federal project directors and planning committees. 

In most cases, evaluation reports of the effectiveness of 
projects are disseminated annually, or at best sem^'^ annually. . 

E. Use of Research and Evaluation Data 

1. Use of Outside Agencies 

Closely tied to the previous topic is the use of research 
and evaluation data. It was found that in most cases Title I 
evaluations are being done primarily by the local school system. 

None of the respondents indicated that evaluations are being done 
entirely by outside agencies. Those systems that do have some 
evaluations done by outside agencies felt the agencies usually had 
contracts in sufficient time to get baseline data about target 
pupils and schools. Usually the outside agencies were not involved 
in the design of the project and did not supply any materials for the 

o 
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2. ’bay-to-Day Project Management 

Opinion^ were split as to whether project evaluation 
have been useful In day-to-day management of projects* Those 
that felt they had been useful said that evaluations facilitated 
continuous modification of the project. Evaluation staffs not 

being large enough to supply the service was given as the primary 

» ‘ * 

reason that project evaluations have not been useful in day-to- 
day project management. 

# ♦ ' 

3. Long Range Policy Making 

It was generally agreed that project evaluations had been 

• ? * * 

useful in making long range policy decisions. Projects are 
continued or dropped and projects are adopted into the regular 

school programs on the basis of evaluations. Evaluations would be 

* • 

more useful in making long-range decisions were it not that 
decisions are sometimes made for political reasons* Timing problems 
also hinder the long range decision making process* When the 
evaluation reports are finally in, it is often too late to alter 
the project. 

• . * « • 

4. Cost-effectiveness Measures 

There has not been much use made of any cost-effectiveness 
measures for Title I programs. Several school systems are attempting 
to use their own cost-effectiveness system and others are in the 
planning stage. 

5. Evaluation Practices 

Both groups agreed that evaluation practices are improving. 
There is currently more emphasis on process evaluations and more 
descriptive evaluation techniques are being, used. The limited size 



of most research and evaluation staffs Inhibits more Improvement. 

The list of **good” evaluation practices sometimes used Included: 
pre-test/post-test deslghlnf^ control/experimental dealgns* and 

randomisation. The research directors also Identified cross- 

♦ 

validation and the use o'f non-obtrualve measures; however* the Title 
I coordinators did not Indicate these. 

F. Staff Adequacy 

Nearly all the research directors said that their staffs 
participate in Title I evaluations. The size of evaluation staffs 
has grown considerably since 1966. The degree of staff turnover 
varies from district to district with the majority of districts 
showing slightly less than a 20% yearly turnover. Most research 
directors reported that their present staff is not large enough 
with the number of additional needed personnel varying* 

G. Budgetary Considerations 

The huge variation in the size of budgets from district to 
district makes an analysis of this information very difficult. 

It is safe to say, however, that all districts desire more money 
for evaluation than they are presently receiving. 
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IV. SUMMARY OF CONFERENCE RESULTS 
Representatives from the member schools of the Council of Great City 

m 

Schools were invited to participate in a conference June 7, 8, 9, 1970, at 
Lake Placid, New York. Participants were chosen from research departments » 
federal program departments. Title 1 teachers, and administrators. 

The purpose of the conference was to gather information about Title I 
projects in the Great City Schools. The research directors and federal 
program directors had already completed the questionnaires described in the 
previous section at the time of the conference. The conference was expected 
to fill in any gaps that might exist in the data collected from the question* 
nairs and site visits. It also was to provide a unique mix of people who 
were knowledgeable about Title I projects in the Great City Schools. The 
wide range of participants was intended to provide for inter*actlon and to 
open new lines of communication within the member schools, and it was intended 
to provide different viewpoints on various aspects of Title I. 

The conference was conducted with both large and small group activities, 
which provided for open interchange of ideas. Participants expressed their 
viewpoints very well. No votes were taken on the different issues; therefore, 
there is no reporting of conference findings in any quantitative manner. 

A summary of conference findings follow. The complete transcript cf the 
conference can be found in Appendix B. 









A. Funding 



One of the most common . concerns expressed by conference 
participants was the inadequacy of funds for evaluation of 
Title 1 programs. If the local education agency is expected 
to measure such things as unanticipated effects and to do more 
process evaluation, then more funds will have to be alloted for 
evaluation. Additional funds are also nestled to enable the 
local district to operate extensive inservice programs to train 
evaluation personnel. 

Along with. the inadequacy of funds, participants -complained 
of the indefiniteness of funds. Funds, they said, were often 
late and the actual amount of funds to be received was sometimes 
not certain at the beginning of the project. 

* 

Some of the recommendations were: 



1. Preliminary funds should be made available upon 
acceptance of the project in general form; with final 
funding to be made upon implementation of the 
project. 

2. A minimum of five percent of tlie project bu^i^et shoruld be 
provided for evaluation. Evaluation funds must be 
increased relative to the complexity of the evaluation 
design. 

3. Additional founds should be provided to allow 
research and evaluation departments to hire additional 
personnel . 

4. Funds should be allocated early enough to allow for 
earlier planning of Title I projects. 



B. Planning 



To assure proper project planning, the participants said 
that an on»golng planning committees should be selected. 



Membership of the committees^ should copsist of administrators* 

$ 

teachers that will be Involved in the project* evaluators* parents, 

and other community representatives* 

Project planning should be started well in advance of the 
proposed starting time for the project. This would allow ample 
time for a thorough Investigation of the situation. 

The planning process should include identification of the 
target populations* using state and USOE guidelines. Once the 
population has been identified* a careful assessment of their needs 
should be undertaken. All available information, both in school 
and out of school, should be utilized in the assessment of needs. 

An appraisal of present programs should then be conducted to 
see if they can be altered to better fit the needs of the target 
population. Along with programs, available resources which could be 
used should be identified. If a new program is necessary, it 
should be designed to utilize available resources as best as possible 
The method of evaluation should be designed at the same time the 
program is designed. By Including a broad range of personnel in 
the planning of the program, it will engender broader support for 
the program. 



C. Decision-making 

Participants expressed concern over the small amount of 
Impact that evaluation reports have had on the decision-making 
process. An urgent need was expressed for evaluations to be 
written in an understandable form. Results should be reported in a 
form useable for the decision maker. This will mean wording the 



reports differently for different levels of decision asking. 

The evaluation staff aust be faailiar with the needs at 

• * 

different levels of decision asking. The experience and training 
of the decision maker at the different levels will to a large 
degree determine the appropriate language for the report. 

The evaluation report must be disseminated widely throughout 
the system and community. Not only administrators but students, 
teachers, community representatives and parents should be kept 
Informed of the results of various programs. The distribution of 
reports should be timed to Impact the decision making process. 

This will require that reports be made more often than on the 
''traditional” annual of semi-annual basis. Evaluation reports 

I 

should be disseminated nationally. Greater use should be made of 
facilities such as ERIC and professional organisations such as 
N.E.A.‘ and A.E.R.A. for national dissemination. 

By making evaluation reports more relative to the needs of 
decision makers and distributing the reports at useful times, the 
research and evaluation staff will greatly assist the decision maker. 
Hopefully the number of decisions made for purely political reasons 
would be reduced. 

It was generally felt that the use of more process evaluation 

* • .* 

would greatly assist the decision maker. More continuous evaluation 
of projects, coupled with immediate* feedback of results would be 
of more value to the decision makers than traditional year end 



evaluations. 



D* Research and Evaluation 



Participants agreed that project objectives should be 
clearly understood by all personnel Involved In the implenenta- 
tlon of the project. It Is most Important that teachers have a 
clear understanding of the project objectives and the methods by 
which the objectives are to be achieved. 

Objectives should be written In behavioral terms in such 
a manner that the extent to which objectives have been achieved can 
be determined. The success of the project should be assessed In 
terms of the achievement of the stated objectives. A variety of 

a 

methods and Instruments would be used to measure the achievement 
of objectives. 

Since the degree of success of the project Is to be determined 
by the extent to which the project objectives are reached^ it 
becomes very Important that the projects are closely monitored to 
eliminate any differences between stated objectives and methods 
and actual classroom practices. 

Participants said that It would be useful to them if a data 
bank of behavioral objectives could be developed. Professional 
educators would write the objectives, with many choices being 
available In each area. The objectives would then be Indexed and 
stored In computer memory for easy access. Persons desiring objec- 
tives for their project could by using the proper Index terms receive 
the behavioral objectives which had been written for that specific 
type of activity. 

It was suggested that research staff members with expertise In 
writing objectives be assigned to each project. The project teachers 



could then receive help in writing objectives whenever necessary. 

♦ 

* 

Conference participants were generally in agreement that more 

process evaluation will be used in the future for evaluating 

Title I projects. Process evaluation was seen as desirable because 

of the continuous self -correction capabilities it offers. 

Concern was expressed as to whether or not present models 

were feasible with present limitations of research personnel. It 

was felt by some that many of the process evaluation models were 

rather weakly defined and difficult to administer. 

Some participants felt that the addition of extra adults 

% 

within the classroom, which is usually necessary to implement 
most models of process evaluation, is undesirable. They said 
it would be better if personnel presently associated with the class- 
room could collect the necessary data required for monitoring the 
project variables. Some people expressed the concern this would 
create a record-keeping overload for the teachers. 

In spite of the many problems seen in implementing process 
evaluation, most participants felt that a growing process evaluation 
"know how" and the recognition of the shortcomings of traditional 

research methodologies would expedite the use of more process 

* 

evaluation. 

. * .> 

Closely associated with the desire for more process evaluation 

is the desire for more longitudinal 'studies. It was felt that the 

impact of some projects cannot be fully measured without some means 

to measure student progress over a longer period of time. 

One problem that was expressed several times was how to measure 

the impact of a project on a particular student, when that student 

is exposed to several special programs. 



It was clear that the School systems are in need of more 
evaluation personnel. With the ever increasing demands being 
placed on evaluation staffs for more and better evaluation reports , 
and the attempt to use evaluation techniques that require more 
people » research and evaluation departmen.ts will have to hire more 
personnel. Where they are going to get the additional personnel 
is an unanswered question, and some training arrangement needs to 
be worked out perhaps with local universities. 

One of the overriding problems is the lack of acceptance and 
understanding of evaluation. Every effort must be made to deliver 
evaluation reports at the proper time and in a language easily, 
understandable for the user. An interesting recommendation which 
came from one of the force field analysis groups was: "Evaluation 

should be included as an essential part of program operation, with 
a five percent minimum budget provided for evaluation. Evaluation 
funds must be increased relative to the complexity of the evaluation 
design." 

% 

E. U. S. Office of Education 

Enough problems were identified with the U. S. Office of 
Education to warrant a separate category. Problems generally 
seem to stem from insufficient two'^way communication between the 
local education agency and the Office of Education. 

Many of the problems expressed dealt directly with guidelines* 
Some participants felt evaluation guidelines which clarified the 
role of evaluation in projects would be helpful. Such guidelines 
could promote process evaluation. 



Present guidelines were seen as too restrictive* sometimes 
conflicting with other guidelines* and changing too frequently. 

It was suggested that guidelines should be written for the 
population they served* that is* a different set of guidelines 
for large school systems. 

Participants expressed several concerns dealing with national 

surveys. The general complaint was that there are too many 

surveys and the information asked for is of little use to the local 

system. The national surveys were generally criticized because 

of poor timing, poor sampling, and poor instrumentation. Many 

. . . • 

people wondered what happened to the information gathered through 
the surveys. They said there was very little feedback to the 
local systems. 

It was suggested that the Office of Education should have 
programs whereby they could offer technical assistance to local 
education associations upon request. 

One of the interesting recommendations was that representa- 
tives from the L.E.A.* S.E.A.* and U.S.O.E stvould cooperatively, 
develop Title I guidelines to be used by the Office of Education. 

Another interesting recommendation was that federal guidelines 

* 

should mandate that funds be concentrated on a maximum of 50 ^ of 



eligible children 



APPENDIX A 



Combined Questionnaire for Research Directors and Title 

Coordinators 

For this project, each Title I coordinator was asked to 
identify two successful Title I projects presently being used in 
their system. To determine the reasons for the success of the 
projects and more effective evaluation criteria, the research 
director and Title I coordinator were asked for information about 
their projects. 

«» 

A questionnaire - interview technique was used to gain inf or- 
nation from the research directors. A similar, but not Identical, 
instrument was given to Title I coordinators. 

In the tabulated questionnaire which follows, substantial 
differences between the instruments have been noted. The response 
from the two Instruments have been shown together for the 
convenience of the reader. Some of the questions asked for responses 
to each of the two exemplary projects. The tabulations, which 
appear as percentages, combine the responses to both projects. 

The ins truments * were developed after on— site visits to two 
of the Great City school systems. The research director and the 
Title I coordinator were Interviewed to obtain the questions used 
in both instruments. These findings do not appear in the tabulation 
of responses. Two additional »on— site visits were made to assist 
in validating the Instruments. 

Responses were obtained from eleven Title I coordinators and 
seventeen research directors, excluding those contacted during the 
two initial on-site visits. 



m 



Where appropriate, ‘responses are given in percent values 

ft 

0 

otherwise, the numbers are raw data. Added material appears In 
italics. 
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THE COUNCIL OF THE GREAT CITY SCHOOLS 

ESEA Title I Project and Evaluation Survey 
Tabulatlorv of Results 



• ♦ 

Coidbined Questionnaire for Research Directors and Title I Coordinators 
1. Successful Evaluations 



(Di7*&otions as they appeared to the research director): 

You have Identified two Title. I project evaluations as being more "respect- 
able" than most. Taking them one at a time» please indicate the reasons you 
believe them to be exemplary. 

(Directions as they appeared to the Title L coordinator) : 

If you had to establish criteria for selecting a gobd evaluation for a Title 
I project, how would you rate the following? 

Select those factors listed below which you deem important to the success of 
the evaluation. Circle: 0 » no importance, 3 ■ greatest importance to the suc- 

cess of the project. If a factor did not exist in a project, circle NA.) 

(Percent valms hccoe been determined in tuo ways* For the coUoms headed 0^ 
Z, 2, 5, DA the percent values are relative to the marber of responses given to 
the question. For the column headed DR (no response) the percent values are 
relative to the total number of possible responses to the question,) 

la. Did the fact that the 0 12 3 NA NR* 

program had clear state- Coordinator % 0 4 22 72 0 0 

ments of objectives Research % 11 15 29 44 0 20 

(i.e», in behavioral Director 

terms) contribute to 
your ability to success- 
fully evaluate it? 



lb. Was a successful evalua- 
tion dependent upon the 
availability of standard- 
ized measurement devices? 

l c. Did the fact that the 
project focused on 
basic skills make it 
easier to do a good 
evaluation? 



Items in this column are percent of 



Coordinator 


% 


9 


14 


47 


28 


0 


0 


Research 

Director 


% 


17 


31 

• 


13 


37 


0 


14 



Coordinator 


% 


not 


asked 


Research 


% 


7 


17 21 39 14 17 



Director 



total responses. 
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0 1 2 3 HA 



Id. Did the fact that the 
project was built 
around a strong research 
design contribute to its 
success? 



Coordinator % 25 • 0 35 AO, _0 

Research % 27 20 31 13 6 



Director 



le. Hhat role did the atti 
tude of teachers and 
administrators play in 
the success of this 
evaluation? 



Coordinator % not asked 



Research .2 0 10 41 48 0 



Director 



If. Can you think of other 
factors which were 
important in this 
successful evaluation? 



1. Wett^trcdned evaluation staff 

2» Utilised oonsultant sewioes 

3. Use of teadher rating scale of teacher process 

d. Parents hired as data collectors 

£. Able to get comprehensive hard data 

d. Used pre-- and post-testing 

7. Distinct separation betaeen control group and treatment group 

8. Used large sample 

In-depth questiomaire given to staff, pi^ils and parents 
10, Use of learning rate - or profile comparison 
Zl. Use of diagnostio evaluation 

12. State and Federal guidelines were helpful, although usually late 
Id. Good computer support (2)'^'* 

14. Jdile to build evaluation obgectives into research design 
is. Good cooperation of program planners 

16. More control in data collection 

17. Successful teacher profile ^ ^ j ^ . 

18. Progect planners and managers agreed on obgectives and evaluation 

19. Teacher observation, behavioral changes, and attendance noted 

20. Some programs are amenable to research, others, noti the design and 
evaluative instruments must be relevant 



** Figures ih parentheses indicate number of responses. If more than one 
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2. Evaluation Problems 



2a, 


As you see It, which of 
atlng Title I programs? 


the following have been major problems In evalu- 
(Check appropriate statements.) 


Coordinator 


Research Director 


• a 


4 




• 

15 


4 


ambiguous project objectives 


10 




12 




Inadequate instruments to measure 

project impact 
• 


1 




9 




evaluation perceived as a threat 


7 




8 




time lag between evaluation and input 
requirements 


2 




4 




unable to employ a competent 
.evaluation staff 


5 




not asked 




inability to effectively cross-validate 
comparable projects 


6 




not asked 




lack of data bank on pupils' socio- 
economic level 






Other 


Comments 



IteeeaiPQh tims utiVis&d fo 2 * move gertevai types of tnfoTmctt'Con <xnd not 
neoessarily evaluation 
2. Inadequate time to evaluate instruments 
S, Too many evaluation requests and demands 

4, One year not enough for longitudinal studies 

5, Too much red tape 

6, Teacher union contract restrictions 

7, Evaluations seen as an *^esstra** 

8» Inadequate cormmioation hetdieen departments 

P. Carnot hire or pay a permanent person from outside of district for federal 
programs 

10 » Difficult to get people to work during the summer when most of the work is 
done 

ll. Need for coordination between agencies demanding evaluation 
12* Discrepancy between the perception of objectives by teachers •and project 
evaluators 

13» Failure to maintain selection criteria for subjects 

14, Inadvertent program change at semester 

15, Inadequate process control 

16, Incomplete data 

17, Meaningful results not given to administration 

18, Inability to innovate on classical designs 

19, Short-range projects with belated funding 

20, Vneertainty of budget 

21, Lack of adequate program design 

22, Broad range of program goals wiVnin projects 
26, Evaluation in the pant tended to be descriptive 

4Z 
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2b* The following examples were cited by a sampling of large city schools 
as effects of a Title I project *hich were not measured and therefore 
did not shot'? up in an evaluation* Check the ones which coincide with 
non— measured effects in your own school system* Feel free to add 
others : 

Coordinator Research Director 



substantial parental Involvement 

changes in parental attitude 

positive changes in attitude about 
evaluation 

reduction in pupil mobility ' 

changes in school administrative 
practices (based on good experience 
in Title I projects) 

improvement in school-communitV 
involvement 

Other Comments 

1, Don*t need professional types measttring us 

2, The effect of cultural enrichment programs 

3, The effect of teacher aides on students 

4, Title I has been effective in getting teachers and principals to plan 
ahead and to think in terms of evaluation and feedback 

5, Negative attitudes build up within non^participating schools 

6, Positive staff development 

7* Modification of teacher behavior 

8, Growth of self-esteem (2) 

9, Social attitudes 



6 

8 

5 



8 

8 

8 



6 

7 



4 

8 



11 



2c* Which of the factors listed below do you believe have either had a 
negative effect on evaluation or by their omission have diminished 
the evaluation effort? (Check appropriate statements*) 



Coordinator 

5 

11 

7 

2 



Research Director 
6 

15 

8 

7 



time of testing 

pupil mobility 

lack of attitudinal measures 

little use of interaction analysis 



o 
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Other Coinments 



1. 

2 . 

3. 

4 . 

• 5 * 

6 . 

7 . 

8 . 
8 . 
to. 

ll. 



Quantity of tasting 

Evaluation reports not welUorganised 

Look of data processing within scTiool system « « . 

Tasting is dona too late to affect progx^ davelopmnt for following year 
Stacking of evaluation demands (i»i», oity^t state and fem^l) 

Hccve to pt(r: and administer as progeot rather than umfved program 

Late arrival of evaluation guide lines 

Overlap in progeots . , _ 

Getting information to deoisionrmakers in useful fom 

Difficulty of follow-up 

General lack of teacher behavior measure 



# 



2d. Rate the various context problems In terms of the difficulties 
presented in evaluating your Title I projects. 

0 1 2 3 NA NR 



teacher attitudes toward 
evaluation 



Coordinator % 18 63 9 9 Q _ jO 

Research % 14 64 14 7 0 17 

Director 



conflict between demands of 
class and demands of evalu- 
ation 

administrators not research- 
oriented 



student unrest 



teacher strikes 



pupil mobility 



sampling problems 



Coordinator % 


18 


22 


18 


27 


13 


0 


Research % 

Director 


15 


53 


30 


0 


0 


23 


Coordinator X 


31 


27 


13 


13 


13 


0 


Research % 

Director 


26 


26 


26 


20 


0 


11 


Coordinator % 


35 


15 


5 


5 


40 


18 


Research % 

Director 


57 


7 


21 


14 


0 


17 


Coordinator X 


61 


5 


0 


11 


22 


18 


Research X 

Director 


42 


0 


14 


21 


21 


17 


Coordinator % 


0 


14 


47 


33 


4 


4 


Research % 

Director 


6 


6 


37 


50 


0 


5 


Coordinator X 


14 


19 


23 


33 


9 


-i 



Research % 20 26 13 33 6 11 

Director 
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Other Comments 

■ ' " ' " I 

J. Lack of dbility to monitop programs on an interim basis' due to ineffective 
measurement techniques 
8, The inertia in programs 

5, Problems of decentralisation ... 

4, Comnunity attitude 

6, Teacher union contract constraints 
6» Lack of manpower 

7, Lack of enough qualified personnel 

8, Lack of clear specification of objectives 

9, Evaluators brought into project too late 



2e. Have you asked for technical assistance from outside sources in 
evaluating your Title I projects? 

Coordinator Research Director 



8 16 Yes 

3 1 No 

If yes, which of the sources listed below did you use?: 

Coordinator Research Director 

Source Not Source Not 



Used 


Useful 


Useful 


Used 


Useful 


5 


4 


1 


12 


12 


2 


2 


0 


5 


4 


6 


5 


1 


5 


2 


0 


0 


0 


3 


2 


1 


1 


G 


4 


4 



Other Comments 

1, Individual consultants (S) 

2, Other universities (2) 

3, Citizens groups 



Useful 

0 local universities 

1 non-profit research 

2 state educational officials 

1 research councils 

0 regional laboratories 
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• • 

«k 

2f. Does the year-to-year funding practice of Title I place any particular 
hardship on evaluation? (Check appropriate statements.) 



Coordinator 


Research Director 




11 


. 14 


o 

Yes» for the following reason (s) 


• 3 


5 

e 


• ft 

hinders ordering of evaluation materials 


4 


8 


places burden on evaluation report for 
re*tundlng 


5 


* 

9 


reduces quality of 'evaluation report 


6 


7 


hinders staff recruitment and retention 




Other 


Cossnents 



1. Requt-pee hupried ^tanning (V 

S, Hinders progrm plcmning (4) ' 

* 3. Progrms for next year must be firmed up b'efore detailed evaluation is 

oomptetedi delays profeat assignments and planning 

Coordinator Research Director 

1 2 No 9 for the following reasonCs) : 

1 0 fosters program modification 

1 1 reduces automatic recycling 

Other Coaments 

1. State allojs several intense evaluations 

2. Aeeountdbility and monitoring stressed 

3. Tends to estdlisk the effective project 

2g. Have you observed school-uide effects of Title I projects on your l.e.a. 
effects which were not stated as objectives of any particular projects? 



(Check 


appropriate statements.) 


Coordinator 


Research Director 


- 


9 


not asked 


tssp for the following resson(s) s 


9 




title I hes become the catalytic agent 
for btoad institutional change 


9 




Title I projects have fostered school/ 
edSBUttity interaction 


6 




Title I programs have been responsible 
for broadening the awareness of extra- 
school responsibilities 

0 



o 
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Coordinator 



Research Director 



2 



not asked 



No, for the* following reason(s): 

» 

Since funding is never a sure thing, 
projects have always operated out of 
the mainstream of the system 

Title I projects have only *' -^hored-up” 
existing programs. They ha ^ never 
been ^imed at fostering widespread 
system change 



3. Successful Title I projects ; 

Your Title I director has identified two projects as examples of -most success- 
ful projects in your school system. Would, you agree with him? 



14 Yes 



0 No 



^ No response 



The t^esearok director and Title I coordinator were directed to respond to the 
following questions relative to the two Title T projects which wer<^ identified 
as exemplary by the federal program director in each system. 



3a. Did the innovative nature 
of the project have any 
role to play in its success? 

3b. Did the fact that ccmummlty 
people were hired to work 
in the project contribute 
to its success? 

3c. Did the project foster more 
community involvement and 
did that, in turn, contri- 
bute to its success? 

3d. Did the age level of the 
target population have 
anything to do with its 
success? 

3e. Was the experience of 

the staff a contributing 
factor to the success of 
the project (i.e., would 
teachers of 5 or 6 years 
teaching experience be 
more likely to succeed 
than the new teacher)? 





0 


1 


2 


3 


NA 


M 


Coordinator % 


4 


13 


27 


54 


0 


0 


Research % 

Director 


6 


16 


23 


46 


6 


11 


Coordinator % 


4 


22 


13 


27 


31 


0 


Research % 

Director 


13 


26 


26 


20 


13 


11 


Coordinator % 


18 


22 


18 


27 


13 


0 


Research % 

Director 


23 


13 


36 


20 


6 


11 


Coordinator % 


9 


18 


13 


59 


0 


0 


Research % 

Director 


23 


10 


13 


io 


2^ 


11 


Coordinator % 


22 


22 


31 


22 


0 


0 


Research % 


10 


3 


ik 


48 


li 


14 



Director 
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3£. Would you say that the 
fact that the project 
staff was Indigenous to 
the coismunity played a 
role in the project’s 
success? 

3g*. How would you rate the 
importance of in-service 
training on the program's 
success? 

The in-service programs were: 



Coordinator 


% 


18 


9 


22 


36 


13 


0 


Research 

Director 

# 


% 


20 


17 


13 


27 


20 


14 


• 

Coordinator 


% 


0 


ft 

0 


31 


63 


4 


0 


Research 


% 


6 


13 


24 


44 


10 


14 



Director 



(Check appropriate statements.) 



Coordinator 


Research Director 




10 


9 


conducted at the beginning of project 


10 


10 


carried on continuously 


11 


12 


taught by school personnel 


8 


7 


taught by outside consultants 


9 


12 


conducted during school hours 


4 


3 


conducted on Saturdays and holidays 


5 


2 


more successful when aaphasising 
socio-cultural understanding 







0 


1 


2 


3 


NA 


m 


3h* Do you think the program 
was successful because 
It adhered strictly to a 
program model? 


Coordinator X 


35 


25 


5 


35 


0 


9 


Research X 

Director 


15 


19 


38 


15 


11 


23 


31 » Did the fact that the 


Coordinator X 


14 


9 


28 


38 


9 


4 


program had clear ob- 
jectives (l*e.» behav- 
ioral terms) contribute 
to its success? 


Research X 

Director 


3 


28 


ii 


39 


10 

• 


17 


3j. Do you feel that the 
project was successful 
because it focused on 
basic skills? 


Coordinator X 


4 


0 


31 


54 


9 


0 


Research X 

Director 




l3 


34 


39 


0 


32 


3k« Rate the importance of 
project leadership: 


Coordinator X 


0 


4 


18 


77 


0 


0 


Research X 

Director 


b 


" i 


ii 


84 


0 


23 



/ 
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Was project leadership: (Check appropria'te statements.) 



Coordinator 


Research Director 


• 

# 


ft 




6 


4 


shared 






8 


11 


full-time 






1 


0 


important only at beginning of project 
« 




11 


13 


important throughout the project 




2 


4 


charismatic 






9 


8 


democratic 






2 


2 


authoritarian 






5 


7 


well-experienced with similar projects 






Other Comments 


• 


1 . 

2 . 

3. 


Humanitarian and flexible 
^oettent rapport with black conmmity 
Competent, thorough adminietration 












0 1 2 3 NA MR 


31. 


Rate the Importance of 


Coordinator % 


not asked 




support personnel to 
project success: 


Research % 

Director 


0 17 10 65 6 14 




Was the staff: (Check appropriate statements.) 




Coordinator 


Research Director 








7 


9 


indigenous to community 




4 


3 


brought in from outside 




9 


10 


experienced in what project was trying 
to do 




3 


2 


had little or no previous experience In 



project alms 
Other Cwnments 

I, Strong union support 

2» Tolermoe for mbiguity due to innovation 

3, Highly motivated 
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0 


1 


2 


3 


NA 


Nh 


3m. How important is it for 
staff to devote full-time 


Coordinator 


% 


0 


4 


4 


90 


0 


0 


Research 


% 


0 


3 


0 


92 


3 


20 


to Title I projects? 


Director 






* 










3n. Do you believe that the 
success of th*ls project 


Coordinator % 


13 


•27 


22 


36 


0 


0 


Research 


% 


40 


10 


16 


30 


0 


14 


: was related to the fact 


Director 
















that coomercially-avail- 
able materials and hard- 
ware existed for project 
utilization? 






• 












3o. What importance would 
you place on the fact 


Coordinator 


% 


4 


9 


4 


81 


0 


0 


Research 


% 


0 


0 . 


44 


55 


6 


20 


that project managers 
must have clear lines 


Director 
















of authority to* Insure 
project success? 


* 






• 










3p. What is the importance of 
community, parent, teacher. 


Coordinator 


% 


4 


A 


27 


59 


4 


0 


Research 


% 


11 


7 


33 


40 


7 


20 


and student involvement in 
project planning to its 
later success? 


Director 
















Should planning groups be: 


















5-10 11-15 16-20 




* 














11 8 3 






■ 












Rate each of the following 
groups according to their 
Importance in project 
planning: 


















teachers 


Coordinator % 


0 


0 


13 


8€ 


0 


0 


Research 

Director 


% 


3 


7 


35 


53 


6 


17 




evaluators 


Coordinator % 


0 


27 


54 


18 


0 


0 


Research 

Director 




10 


7 


:^9 


42 


0 


17 




parents 


Coordinator % 


5 


25 


45 


25 


0 


9 

T7 


Research 

Director 


% 


32 


39 


i7 


10 


0 






students 


Coordinator % 


AO 


0 


35 


5 


20 


9 

17 




Research 


% 


6k 


7 


14 


7 


7 



Director 




m 
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school administrators 



consultants 



3q. How important was process 
evaluation to the success 
of the project? 



Coordinator 


% 


9 


0 


22 


68 


0 


0 


Research 

Director 


% 


6 


10 


32 


57 


0 


17 


Coordinator 


% 


18 


4 


13 


45 


18 


0 


Research 

Director 


% 


21 


35 

• 


21 


14 


7 


17 


Coo: dinator 


% 


19 


9 


52 


14 


4 


4 


Research 


% 


16 


25 


16 


16 


25 


29 



Director 



(Check appropriate statements.) 
Coordinator Research Director 



1 0 

10 9 

6 7 

8 8 

3 4 



process evaluation is only worthwhile 
during the first year of the project 

♦ 

process evaluation should be as important 
throughout the project as it is during the 
first year 

process evaluation should be done by 
project managers 

process evaluation should be done by the 
research and evaluation division of the 
school district 

process evaluation should be done by 
management consultants (from outside the 
school system) 



3r. Other factors important to success of the project: 



1* WeVl^tmimd teachers 

8. Creative idea initially 

S, Time for greater teacher creativity 

4, Concerned with mador social problems 

5. Evaluation should be used for progrm dedsion-^rndking 
€, Prospective long-term achievement gains 

7, Dedication of staff and teachers 
8» Acceptance of proposal by teachers 

9, Organisational structure that allows clear comunioatim with personnel - 
horizontal and vertical 

lQ» Newsletters, papers, radio 

11, Linkages with this project and others 

12, Adequate funding oomitment 

13, Interim reporting on the progress oj the project 

14, Preschooling md parent participation 

16, Interchange of ideas, organisational patterns, professional growth 

16, Team leaders free of classroom aosigments 

17, Support of VI omujping agencry. 



fit 
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A. Project Changes 

Aa. What *is your feeling about changes in your l.e.a. *s Title I projects 
over the years? (Check appropriate statements.) 

Coordinator Research Director 



0 

8 

A 



3 projects, for the most part, have not 

changed . 

7 there have been gradual but no dramatic 

changes • 

A changes have been so significant that pro** 

grams today bear little resemblance to 
earlier ones. 

Other Coicments 



t. Strongly affected by deoentraHsation 

2, Changes not due to program needs but to state and national pressures 
8, New prooeots haoe been funded to meet needs 

4, Consistently increasing focus on basic skills 

5, Changes have been graSial and significant 

6, Some as a result of guidelines 
7* Upgrading assisting programs 

8* More concentration of resources 

Ab. If you believe changes have taken place in your l.e.a. *s projects in Title 
I over the years, check items below which indicate the nature of that 
change: 



Coordinator 


Research Director 




7 


6 


more emphasis on child growth and develop- 
ment (whole child) 


9 


8 


more emphasis on basic skills 


10 


11 


more emphasis on early childhood education 


1 


3 


away from affective behavioral changes 


5 


not asked 


from supplemental (after-school) to regular 
(during the day) progratiB 


8 


not asked 


more continuity programs for target pupils 


A 


not asked 


greater concentration on nutrition needs 


6 


not asked 


more emphasis on innovative programs 



IBStt 
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• • 

Other Comments 

Z. Concentration of effort on smaller group 

2, More parental involvement 

3, Need more convincing data to gain confidence to stop or suspend progects 

4, Because program lias been highly sound ond suooessfut, drcsna.tic changes would 
cost a great deal more 

5, Clear concise reports should be required by Federal government^ to avoid 
unnecessary collection of data and duplr cation of effort 

6, Lack of understanding of potential usefulness of feedback 

7, Nature of funding proves something to do with it; certain funds are ear-^ 
marked], state allocation system hinders innovation 

Rather see gradual ohange 

4c. Who makes project management decisions? To what extent? (Check 
appropriate column.) (Coordinators not asked this question,) 



m 

project manager 
federal project director 
project evaluators 
support personnel 
planning committee 
program innovator 

4d. When Is data from evaluations on project effectiveness disseminated? 
(Coordinators not asked this question,) 

1 weekly 

1 monthly 

5 semi*annually 

7 annually 

Other Comments 

I, Frequent pvooess evaluations; some q;uarterly, some even daily. 



Research Director 



Most 
• 11 

4 

0 

0 

2 

0 



Some 

3 

4 

5 
3 

6 
2 



Rarely 

0 

3 
5 
5 

4 

5 



5. Use of Research and Evaluation Data 

5a. Are your Title I evaluations done by outside agencies? (Coordinators 

not asked this question,) 

Yes 6 No 9 

88 
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VHiat percentage? 

10% 2 • 20% 1 30% 1 50% 2 75% 1 

Are they contracted for In sufficient time to get baseline data about 
target pupils and schools? 

Yes 5 No 3 

Were these outside agencies involved in the design of the project? 

Yes 3 No 5 

Do they supply any materials for the project? 

Yes I No 7 

5b. Have your project evaluations been useful in day-to-day management of 
projects? 

• a . , 

Coordinator Research Director 

5 6 Yes, for the following reason (s) : 

5 3 Facilitates continuous modification 

Other Conments 



Z. Uot as moh as should 

2, Striding to improve 

3, Diagnostio in nature^ smalt staff limits 

4, On-site evaluation used for day^to^day program modification 
6, Not until recently 

6, Some in-house data availcible at plaming time 

7. Amendments make this possible 
3. Facilitates octmunity awareness 



Coordinator 


Research Director 




6 


11 


No, for the following .reason(s): 


2 


1 


* When budget is set at beginning of year 
little can be modified 


5 


7 


Evaluation staffs not large enough to 
provide service 
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Other Conanants 



2, Muck data has not been put to use 

2. Need better utilisation of data to be effective 

3, Gross missing of project objectives 

4^ School system regulations limit change — materials and shift of personnel 

5. Useful at points in time when project manager can affect changes (e,g, at 
'semester) 

6» Se^-annual reports oan*t do mych dbe-ft day-^to^day problems 
?, Timing of evaluations bad 

B, Hardware deficiencies (no emputer facilities for feedback) 

9, Understaffed to collect data (2) 

20, Lack of realistic requirements from evaluators 

21, Not enough funds 

12, Title I programs have been autonomous 

23, Insufficient intemediate goals 

24, Program design set-up 

IS, Admendment must be submitted to State Department 
IS, Full report not avoLldble at planning time 
27, Little effect on individual teacher 

5c. Have your project evaluations been useful in making long-range policy 
decisions? 

Coordinator Research Director 



11 

9 

8 

5 



10 Yes» for the following reason(s): 

6 Projects are continued or dropped on the 

basis of evaluation 

6 Projects are adopted into regular school 

programs on the basis of evaluations 

3 Evaluations have affected school-community 

relations 

Other Comments 



2, Project modified to indicate evaluation direction 

2, Both adversely and positively 

3, Usually with a six-month time lag 

4, Used as corroborative evidence for hoard decisions 

6, Not as much as they should 

5, Usually takes 2 or 3 years 

7, Evaluation data is used in decision making process 
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Coordinator. Research Director 



8 

6 

2 



2 

6 



4 

3 



No, for the following reason(s) : 

(Coordinator^ not asked this question,) 

Decisions are made for "political** reasons 

Data in evaluations is usually trivial 
and can't be used as basis for decision- 
making 

Never gives information on cost-effectiveness 

Timing problems: When the results are 

known, it is too late to alter project 

No way to isolate effects of Title I 
activities from otljer school and community 
experiences 

Reports should be clarified in terms of 
decision alternatives 

Translate reports into language under- 
standable to decision-makers 



Other Comments 



1, Evaluation vesearoh is not top priority with high adtdnistrators at decision- 
making levels 

2, Not much long-range planning 

3, Changes cost money; insufficient cost-benefit analysis 

4, Federal guidelines unstdle 

5, Reports not in language comprehensible to decision-makers 

6, Politioal problem occupying attention of high administration 

7, More consistent evidence would force long-range planning 

8, Research considered high enotigh level 

9, Pushing for process evaluation 

« 

5d. Have you used any cost-effectiveness measures for Title X programs? 

If so, what are these measures? 

1 PPB System 

0 REAL Model 



Other Comments 

1, Months gain/pupil cost 

2, Cost per pupil relative to normal per pupil costs (2) 

3, . Costs of retention in grade/savings 

4, Making awn model 

6* Line item analysis S6 

S6 



o 
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5e. Are your evaluation practices iiaproving? 



Yes /for the following reason(s)j 

more Quasi-experimental designs 

mc<;e non-obtrusive evaluation 
techniques 

more descriptive .evaluation techniques 

more emphasis on process evaluation 

increased ability to identify control/ 
experimental groups 

Other Comments 

* 

1. GreateT staff skilt through praotioe 

2. More attention to behavioral objeotives ^ 

3. The use of greater understanding of various models 

4. Itnpraved language 

5. More teahnioal t^upport 

6. More ooimtunioaiion with deoision makers 

?» "Evaluation personnel involved more in project planning stage 
d. The addition of student follow-up 

9» Better leadership ^ t • j. jfj* 

tOm Better ooordi<T.ation between planning ^ management . end evaluation staffs 



Coordinator 



Research Director 



10 

.5 

5 



12 

4 

e 

4 



5 

7 

2 



9 

9 

5 



Coordinator 

1 

0 

0 

1 



Research Director 
7 
2 

2 

4 



No, for the following reason (s) : 

inability to modify classical 
designs 

can't get qualified staff 
staff too small 



Other Comments 



7. "Program changes have required starting over each year 
2» Unreasoncdflc demands for several Federal evaluatiotut 

5f. Which of the following do you consider to be good examples of 
evaluation practices used in your school system? 
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Coordinator 


Research Director 


10 


16 


9 


12 


5 


8 


1 


9 


2 


11 


0 


1 



pre-test/post-test 
control/experlmental 
randomization 
cross-validation 
non-obtru.iive measures 
simulation experimentation 
Other Comments 



1. Classroom observatioizs 

2, Development of non~oulturat instruments 

5. Proaess evaluation and system analysis 

4, Outlining of probable outoomes of various proQeot alternatives 

5, Case studies^ sudk as of most and least successful units 

6, Longitudinal folloD^up 

7, Careful analysis of most successful and least successful units in program 
Observation of classical practices by qualified consultants 

9, Progect description workbook 

10, Training administrators in specification of objectives 

11, Outlining of project alternatives 

12, Repeated measures technique 
IS, Case studies 



6, Staff Adequacy 

6a. Does your staff participate In Title I evaluations? 

(Coordinators not asked this question,) 

16 No 1 

Of your total staff, how many participate in Title 1 evaluations? 

During the period from 1966^1969 ^ there has been a tremendous increase 
in the size of evaluation staffs, with the increase in staff ranging 
from two times to thirty times as many, 

6b. Is there a heavy turnover of your full-time staff: 

(Coordinators not asked this question,) 

Yes 7 No 8 



What degree of turnover? 

The majority of responses showed less than 20% turnover. 
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6c. Is your staff large enough to meet current needs? 

(Coovdirtatora not asked this question,) 



Yes 2 



Ne 14 



If not, how many more are needed? 



1-5 

6 



6-10 

2 



11-20 

2 



21-up 

1 



7. Budgetary Considerations 

7a. How many dollars did your school system put Into evaluation this year? 
Of that, how much came from Title I funds? What is the total l.e.a. 
budget? 

(Coordinators not asked this question,) 

From $44 >000 - $2,000,000 Total for evaluation 



From $60 million - $1.6 billion Total LEA budget 

7b. What percentage of the budget do you think your school system should 
use for evaluations? 



From $12.000 - $1.200.000 



Title X evaluation expenditures 



1 - 2 % 

5 



3-5% 

5 



10 % 

1 



20 % 

1 



50% 

1 






o 
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APPENDIX B 

Lake Placid Conference 

The Lake Placid Conference was designed to obtain the Inputs from a 
unique mix of people ivivolved in Title I. The participants were representative 
of the following categories: 

1. Research and Evaluation 

2. Program designers 

3. Administrators 

4 . Teachers 

Dr. R. W. Napier was retained to direct the group activities during the 

• • • • 

conference. He varied the activities between large and small group exercises. 

The emphasis was on individual participation within the groups. 

The conference was divided into four sessions. Each session was designed 
to contribute information about a general topic within Title I project design 
and/or evaluation. 

The conference was acclaimed a success by the participants and the Coqncll 
staff members. The reasons for its success were twofold. First, the unusual 
combination of Title 1 people opened up new channels of communication within 
the Great City Schools. Second, the participants were very knowledgeable 
about Title I prograins and were eager to share their experiences with their 
colleagues and with the Office of Education. The success of the conference 
was also due to the masterful way in which Mr. Napier worked with the participants. 

Each group was assigned a specific task and asked to record their data. 

The information was compiled from each of the four sessions. 

The problems, solutions and fiascoes have been arranged in the broad 
categories of research and evaluation design, planning, deals lon*>making, 
personnel, and Office of Education, to facilitate content analysis. 

o 
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Session I 

m 

Problems. Solutions and Fiascoes 



This session was designed to warm*>up the participants* The participants 
were divided Into small groups and asked to discuss creative solutions to 
evaluation problems and evaluative fiascoes with which they were familiar* 

The groups warmed up quickly to their task and produced much information* 

A subsequent analysis of data showed they produced mere examples of problems 
than solutions* The information received from Session I follows with no 
interpretation* 



Research and Evaluation Design; 

Problems with Accompanying Solutions 



1* Problem 

. Solution 

2* Problem 
Solution 

3* Problem 
Solution 

4* Problem 

Solution 
5. Problem 
Solution 
6* Problem 

o 



Evaluation time lag between report and new program 
submission* 

Minimum of 3-year funding to provide feedback of 
evaluation data Into proposal development* 

Variety of measurements of program* 

Increased use of non-obtrusive techniques in evaluation 
such as vandalism, and school achievement plotting. 

Time inadequate to test effects* 

Longitudinal 4th and 6th grade effects on students in 

4 different school, settings* 

I 

Application of statistics which have a basis of normal 
distribution to groups which by nature of Title I are 
not on the same norms across any given variables. 

Use self-norms* Movement should be from the base of 
the pupil's achievement* 

Same child exposed to more than one Title X program; 
inability to determine impact of a special program. 

Control charts which identify the number of programs 
which have been used at a school* 

Dlfferlnj» objectives locally within one program. 



Solution 
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Broad system-wide objectives with more specific local 
objectives as program gets nearer the point of action. 
Example: lead teachers program - research associates 

link th^. central system to schools with lead teachers. 
Improved communication feedback through process evalu- 
ation - feedback every 2 months, redirecting program 
as necessary. Made program visible and accountable. 



Problems 

,1. Detecting the difference between stated objectives and real practice. 

2. Assessing behavioral objectives. 

3. Nationally-defined samples do not Include schools adequately, i.e. , 
target area schools have varying number of target pupils. 

4. Should the results be in terms of *'raw scores,” "grade placement,” 

stanines and "percentiles," all reporting the same results? Is . 
there growth when only 2 points in raw scores are gained, which may 
show .5 grade level growth? How do you interpret this datum? 

5. Disagreeraenj; over reading levels - standardized score not agreed upon 
as a behavioral objective. 

6. Timing for evaluational testing. Test conflict with regular program. 

7. Sampling problems: Experimental/control grouos. Attrition, mobility. 

. contamination, difficulty of matching. ‘ ooixicy, 

8. Researchers not satisfied with ability to monitor different versions * 
of city-wide projects. Can evaluator pinpoint project's effects in 
experimental designs? 

9. How much change is ethically acceptable in deviating from original • 
goals? 

10. Cognitive vs. affective goals - or social service goals, e.g. food, 
medicine, etc. 



11. Dichotomy between avoiding narrow, restrictive statements of objectives 
to satisfy legislators* demands, and the fact that there must be a 
baseline of clearly-defined goals. 



Solutions 

1. Some schools with remedial reading programs found 1 year, 4 months gain, 
but cause i^ras undetermined. 

2. Establishing . learning curves. Data banks through computerization. 

3. Checked several schools* character ifstics pre-ESEA in reading readiness. 
Revo.'iJed 1 month advance in reading level compared to city-wide decline* 
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4. Customized testing around specific objectives. 

5. Learning objectives prepared by professionals! catalogued and stored 
in a computer for easy access » with thousands of choices in each area. 

6. Learning rate prediction » based on nuinber of years In sdieol miltl* 
plied by achievement place * number of months gain per year. 

7. Use of in-class observation by parents during the year to show evi- 
dence of learning. 

8. Trend evaluation being accorded more credence by the advent of process 

evaluation. Longitudinal studies and greater support data is needed to 
support rationale. 



Flascoes 

1. Isolation of treatment variables. 

2. Questionable data used. 

3. Forced evaluation without valid evidence. 



Planning ; 



Problems with Accompanying Solutions 



1. Problem 



- How to get raw data of test results back to schools so that 
planning and Implementation may be related to these results? 



Solution - Data bank processes Immediately, sends back to school and 
program director. 



2, Problem 



- How to get overlapping progr^s and related services to get 
and use Information? 



Solution - Inter-Dlsclpllnary Questionnaire, designed by researcher 
that pointed out many services and their Impact, showed 
each service what the other was doing, then suggested they 
develop some way of sharing information. 



3. Problem 



- How to develop continuous performance objectives as a basis 
for planning and Implementation? 



Solution - Assign a researcher to project to work directly with parents 
and teachers. Train teachers in methods of writing per- 
formance objectives as a basis for evaluation, continuous 
self-direction, and change of goals. 

4. Problem - Feedback of test results. 



Solution 



Test results turnaround time (1 month) given to classroom teach- 
er with output of test scoring by classroom teacher now possible. 

633 
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1 . 



Hbw to use evaluation results 



for planning and/or implementation? 



Dec is ion-making ; 



Problems 

1. Programs cannot be eliminated solely due to inability to measure effects 
of programs. 

2. In f,ome schools evaluation has complete freedom to ”tell it like It is” 
unlike, some districts which are pressured to report ”good*' results , 
while others look for failures to give decision-makers. 

3. How much improvement in reading or math .justified removal of funds from 
art» outdoor education^ etc.? 

4. External decisions, e.g. transfer of teachers under court order on* 
desegregation. 

5. Greater initiative needs to be mutually taken by school, community, and 
federal levels to Improve guidelines for program development and theory. 



Problems with Accompanying Solutions 

- Eliciting cooperation of school personnel. 

- Direct involvement in developing project plan and evaluation 
design (with evaluators as consultants). 

- Role perception - Aides completing student ratings. Teachers 
viewed it as "diagnosis" and were threatened. 

- Employ graduate students to reduce teacher-aide tensions. 

- Feedback to project and school personnel. 

• 

- Interim reports, continuous assessment and reporting to teachers 
and principals' workshop with summer reports by project leaders 
and with discussion (reports presented by evaluators). 

4. Problem - Affective domain involving teachers. 

Solution - Training groups composed of differences between teachers to work 
in inner-city and outer-city. 

5. Problem - Lack of continuity in evaluation due to change in personnel. 

♦ 

Solution - Each progtam should have at least 2 evaluators, 1 person with the 
major responsibility for evaluation and the other with minor 

64 
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Personnel ; 

1. Problem 
Solution 

2. Problem 

Solution 

3. Problem 
Solution 
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responsibility, leaving two people with knowledge abdut the 
program who can benefit from each others' counsel* Replace- 
^ ment of* evaluators who may leave would insure continuity of the 
evaluation process* 

• % * 

6* Problem - Status of paraprofessionals; paraprofesslonals who want to help 

children learn often leart: more about 'helping children than does 
the teacher, causing conflict. 

Solution — Instructional Paraprofesslonals Handbook developed by represents"- 
tivea from each ESEA school, with an addendum geared to each 
school site* Self-evaluation and teacher evaluation of aides 
matched* Developed aide leadership and understanding of ESEA, 
and made administration of aide effort more efficient and effective 

Problems 

1* The tremendous volume of data required for various external agencies* 

2* Teachers with access to evaluation instruments geared their teaching to maxi- 
mize student achleverodnt* . . 

3* Change in content of the message as it filters through the ranks. 

4* Inability to get teachers to fill out questionnaires of evaluators, because 
teachers do not understand purposes of evaluation, nor benefits to them- 
selves. Conflicts with regular classroom program, especially at end of school 
year. Growing teacher milltance* 



Office of Education ; 

Problems with Accompanying Solutions 
1* Problem - Multiplicity of required evaluations* 

Solution - Belmont, CPIR, Carl Perkins (Congressional Committee) 

Problems 

1* Need definition of evaluation at different levels* 

2* Clarify X.e*a*, s.e*a* roles and responsibilities in evaluation* 

3* State and OB evaluations are meaningless T 

Flascoes 

1* National Advisory Committee on Disadvantaged Children evaluation report. 

Not based on legislative purposes as stated in ESEA - BESE not sufficiently 
involved. Some disagreed, not critical of Committee report. 
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Seiaton II * 

Probleias In Evaluating Title I Pro , -|ecta 
In this tesslon, the participants were divided Into small groups. 

Each grcK'p was given the task of Identifying problems in evaluating 
Title I projects. This session was very productive lu terms of the 
number of problems Identified. The data from this session follows. 

Research and Evaluation Design ? 

1. Provide resources to match evaluation expectations 

2. Emphasis on hard data (tests and behavioral objectives are too 
restrictive) 

3. Too many objectives for project 

4. Frequently changing goals 

5. Frequently fuzzy goals 

6. Lack of evaluation of non-students (e.g., organizational changes, 
teacher changes, community changes — attltudlnal) 

7. ' Evaluating the proposal as Impacted by funding 

8. Limitation of strategical methodology (too many evaluative problems) * 

9. Identification of target population 

10. Tendency to separate evaluation from program operations • 

11. Role conflict between operation and evaluation design 

12. Continuing data collection as part of evaluation process 

13. Excessive testing of children 

14. Multiple project effects on children 

15. Insufficient baseline data 

t 

16. Lack of emphasis on longitudinal dimensional problems in carrying 
out longitudinal studies 

17. Instrumentatlve measurement of process 
18* Normlng of measures 

19. Mob.lllty of. pupils 

66 . 
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20* Impact of assessment methods * 

* 

21* Local verification of replication 

22* Unrealistic time demands contained within the evaluation 
design 

Planning : 

1* Lack of sufficient theoretical base to Title I 

2. Lack of coordination of research expectations limits plan- 
ning involvement 

3* Failure to include a "significant meiQber" of evaluation in 
planning 

4. No coordination of research with rest of organization 

5. No connection with decision-making process 

6. Evaluation separate from program planning 

7. Insufficient lead time for planning and evaluation* building 
instruments* and collecting data 

8* No consistent data collection procedure 

9=» Unnecessary duplication of data gathering among various projects 
Decision-Making ; 

1. Need to Interpret results in understandable form 

2* Lack of differentiation of evaluation design to n^et the needs 
of various decision-making levels (from classrooms to Washington) 

3. Political decision-making at local and other levels 

4. Decision-making improvements 

5. Role of evaluation: development or decision-making 

6. Urgency of proving point 

7. Emphasis on positive results 

8* Identification of the real decision-making process 

9. Failure to Indicate the -ervlces to be Implemented 

10* Lack of commitment on part of decision-making to use of eval- 
uative feedback 
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11. Ineffective dissemination of findings dnd conclusions from 

program evaluation ‘ . 

12. Non—relevo*'.' of upper echelon data collecting efforts 

13. Lack of LDntation of school personnel regarding Title I 
involve iiiit 



Personnel : 

1. Who are the prime users of evaluation reports? 

2. Lack of acceptance and understanding of evaluation 

3. Resistance to innovative measurement ideas 

4. Project staff confidence in evaluation 

5. Educating teachers and administrators to evaluation as a means 
of improving programs 

6. Limitations of training personnel 

7. Time burden on teachers and administration in development and 
operation of evaluation 

8. Lead teacher time for technical questionnaires 

9. Integrity of evaluation 

10. Lack of profess %lly trained evaluators 

11. Unclear role relationship of evaluator 

12. . Unclear lines of authority 

13. Independence of evaluator 

14. Stability (flux) of personnel at 0£ 



Funding ; 

1. Indefiniteness of federal funding, i.e., received too late in 
school year and uncertainty of actual amount 

2. Distributing funds hindered by need to identify, concentrate 
on target pupils 



3. Inadequate budget 

4. Inadequate funding for process /management evaluation 



Evaluation of fiscal data 
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6* Political implication of a really successful program because 
it might cost too much for local implementation 

Office of Education ; 

1. Lack of evaluation guidelines 

2. Frequent changes in guidelines 

3. Guidelines too restrictive to allow sufficient local flexlb~ 
ility 

4. Inadequate budgetary (funding) guidelines for evaluation 

5. Guidelines do not promote process evaluation 

6. Guidelines differences at Federal, state, and local levels 

7. Guidelines geared to small district administration and evalua- 
tion . . 

8* Clarification at national, state and local level of Title X 
research and evaluation expectations 

9. Local programs do not generally match national objectives 

10. Priority of 0£ objectives over local objectives 

11. Criteria used to select school for OE survey 

12. Poor national sampling (re: national survey) 

13. Proliferation of, and non-coordination of (national) surveys 

14. Timing of survey bad 

15. Lack of representation of local districts in developing survey 

16. Poor Instrumentation for national survey 

17. Not enough use of self-evaluation by the national team 

18. Insufficiency of local consultants 

19. Inadequate data-gathering techniques for presentation to Congress 

20. Lack of technical assistance frmn the federal level 

21. Where is feedback of OE evaluation at program results? 

22. Lack of confidence of federal level in the evaluative ability 
of local districts (mutual credibility gap) 
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Session III 

• Case Studies 

The Council stalf wrote four brief case studies. These case studies 
were used to set the scene for participants to design a Title I project. 

Two small groups independently developed their own projects for each case 
study. Each group was free to make its own assumptions about the case. The 
case studies with project designs follow. 



Case Study Submitted to Work Groups 
. Self-Esteem • . 

I. Short Narrative Description 

A frequent observation of many educators is the fact that pupils just 
don t try hard enough to achieve in schools. Explanations given as reasons 
for this fact) especially as it applies to large nusd>ers of inner-city pupils » 
is that this is symptomatic of a low self— esteem^ and lack of reinforcement for 
scholastic accomplishment in the home. Numerous techniques have been attempted 
which were intended to **raise” this low self-esteem of certain public school 
children. The following ESEA Title I program is a hypothetical example of an* 
intervention technique which is supposed to attend to this problem of raising 
self-esteem in an inner-city target area school. 

4 

An interested citizen in a local city has had extensive background in the 
theater. This particular citizen is convinced that by encouraging pupils to 
pantomime certain actions described in poems > stories, and related skills 
subjects, these pupils will exhibit a demonstrable increase in their self 
esteem and scholastic achievement. Upon impressing the practicality of this 
approach on certain influential public school policy-makers, the interested 

•4 ‘ . • 

citizen is encouraged to develop a program, funded through ESEA, Title I monies, 

70 
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The first-year operation of this program is to take place in five 
target area schools fri^m which our interested citizen will personally form 
dramatic clubs, composed of children cohsidered by their teachers as being 
verbally and socially inhibited, A staff development component is included 
in this program, and these teachers meet with the directoress of the program 
These teachers are paid for their time, and are not necessarily the same 
teachers whose pupils are Involved in the program. 

The evaluation of the first year’s operation is in the form of teacher, 
questioniiaires. Only those teachers involved in staff development programs 
are obligated to t.ake these tests. 



II. Outline of Objectlvds 

A. To enhance the self-image of target area pupils through the medium 
of creative crama 

B. To introduce the concept of creative dramatics as a viable teaching 
^ technique, to the professional staff 

T p * 

C. To increase pupil scholastic achievement as the result of their 
experiences in creative drama 



D. To improve the self-confidence and social-confidence of pupils as 
the result of their experiences in creative drama. 



Project Designed from Case Study 
Self-Esteem 
Group I 

Assumptions ; 

!• Planning grant 
4-6 months 

Involving - administrators 
teachers 

. community 

Goal r Develop "their program” 

^4 * 
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2. Staff development program for administrators, teachers, community 
personnel to provide for training in *developinj? of performance 
objectives to be used as a basis of evaluation 

m 

3* General orientation for all school personnel involved with this 
project 

4. Teachers who are involved in design will be the teachers who operate 
the program. 

5. All facilities will be available when program becomes operational . 

6. Design of evaluation model and operating program will be concurrent. 



Evaluation Design ; 

1. Series of teacher, parent, administrative questionnaires — both 
oral* and written 

2. Utilize existing standard scholastic achievement test, additional tests 

will not be given. * 

3. Base line data 



Project Designed from Case Study 
Self-Esteem 
Group II 



Assumptions ; 

1. Interested parents, teachers, administrators, and evaluators were 
involved in the development of the plan. 

2. Set up criteria for identification of eligible pupils. 

3. Identify factors which determine degree of verbal and social inhi- 
bition. 

4. Staff responsible for implementing program, develops behavioral 
goals . 

5. Staff development component to Include only those teachers whose pupils 
participate in the program. 

6. Need to build a data base 



* 



TV interview programs to be utilized as a dimension of evaluation using 
this kind of expertise. 
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Find out if student participated in other programs to know of 
any cross-pollination effects » 

Needs assessment — find out as much as possible about each 
participant) both in and out of school* 

Build program based on what we know of each as human beings. 
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Case Study Submitted to Work Groups 
Reading Program 

X* Short Narrative Description 

The Spring City Title 1 Planning Committee is meeting to plan the second 
year of their Elementary Reading Project* The project was lAitiated the pre- 
vious year because the Planning Committee had determined that the highest 
priority in assessing the needs of the Clifton Avenue School he assigned to 
improving reading. Testing programs had revealed that 90% of the largely 
black target population were reading at levels 1 and 2 years behind grade 

level. . . . 

The main elements of the»project consisted of offering to the most edu- 
cationally deficleut readers in the 2ndp 3rd and 4th grades in the shhool a 
supplemental reading Instruction period; and working with other school re*- 
sources such as psychologists* classroom teadier and nurse to detect individual 
difficulties. 

IX. Outline of Objectives 

The objectives were: to improve the level of reading for each student; 

develop improved attitudes toward reading; increase the nuniber and variety of 
reading matter and build respect for the care and handling of books. 

The coisaittee consisted of a reading teacher* project evaluator* school 
principal* project manager and parent 'of a student in the project. 

Project Designed from Case Study 
Reading 
Group I 

Research Design for Reading Program* Experimental Program: 
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I. Diagnostic Reading Program. to Determine Specific Problems! 

« 

A. Reading grade level 

B. Aptitude level 

C. Possibly physical, psychological or cultural handicap 

D. Review of instructional reading program 

1. Strengths and weaknesses of staff 

2. Instructional materials 
II* Special Experimental Activity 

m 

A. To establish a reading learning center at school with emphasis on 
staff developing a variety of instructional approaches to reading. 

1. Develop parental involvement by employing indigenous non-profes- 
sional personnel. 

III. Objectives 

A. To improve the child rate of growth in reading to double his current 
level within the period of one school year. 

B. To explore alternative methods of program approaches to increase 
pupils’ reading leve.ls* 

1. Individual Instruction 

2. Phonetic emphasis 

3. Use of teaching machine 

4. Student tutor 

5. Paraprofessionals 

6. Reading learning center 
IV. Evaluation of Program 

A. Continuous progress evaluation of each child 
1. Process evaluation type 

B. Supplementary use of established standardised test 
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Project Designed from Case Study 

« 

Reading 
Group It 

Assumptions 

1. The committee is on*-going since the inception of the project. 

t 

2. This committee generated the original proposal. 

3. The committee accepted input from other people, including partici- 
pants in the project. 

A. The committee analyzed and re-analyzed the objectives of the first 
year. 

5. Workshop held at the end of the first year with those involved in 
the program — to change the processes and to define an evaluative 
model. 

6. Time and money was available for participants in these review and 
planning situations. 

7. Specific criteria were used to identify most needy kids for inclu- 
sion in the project, and for identification of the instrumentation 
to be used. 

8. Appraisal of resources available; e.g. the number of reading teachers. 
Assume that part of the design of the second year is to measurr vari- 
ables contributing to student gains; e.g., individualized instntction, 
resources, and teacher style. 



Component Design 

1. Identification, notification and returned acceptance from those 
personnel selected as teachers, administrators, etc., in project. 

a. Time to develop and order new instruments 

2. New behavioral objectives, or redefined objectives — with enough time 
to define them. 

a. Including participation of planning committee and significant others 

3. Time to establish criteria for student participation. (Some kids may 
”achieve-out” . ) 

* 

A. Time to assign controls 

5. Design and identify persons to be involved in implementation with 
various responsibilities in feedback (as information is generated) 
and schedule as to when feedback is to be generated. 
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6. Enough funding to hire evaluation consultants , and flexibility to 
adjust the evaluation program in process in anticipation of the 
succeeding year;* also to develop training of evaluation personnel. 

7. Ongoing contact, and information exchange with other relevant reading 
projects. 

8. Availability of subject matter consultants and research experts from 
the local university. . 

a. Follow-up of those students who dropped out of the project at 
the level of the first year. 

9. Foreknowledge of contributions to be demoded of project in re: 
federal and state surveys. 

10. Continuous process evaluations. 

11. Time to develop evaluative criteria in behavioral terms — defined in 
one month increments. 

f • • 

12. Restructuring the program based on results of evaluative criteria. 

13. Develop a research design which will trace causal relationships 
between input and achievements; also a variety of measurements 
comparing the student vs. himself and the student vs. other students. 
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Case Study Submitted to i^Tork Groups 

« 

Drop-Out Prevention 

I. Short Narrative description 

Jefferson high Senool has been identified* as within the target area for 
the district's Title I program* 

Over the past four years, the drop-out rate at Jefferson has been more 
than 42%. The Title 1 planning committee has decided upon a course of action 
\«hich hopefully \^ill reduce the drop-out rate significantly at Jefferson high 
Scnool. 

The drop-out reduction program will consist of three basic areas of con- 
centration: , 

(1) Communication 

(2) Mathematics 

(3) Vocational Counseling 

II. Outline of Objectives 

1. To substantially reduce the drop-out rate of Jefferson High School 

111. Specific Objectives 

1. To improve the communication skills of the target population. 

2. To improve the math performance of the taiget population. 

3. To give students an understanding of the manpower needs of local 
businesses. 

4. To make students aware of the training and educational requirements 
in various vocations. 

Project Designed from Case Study 
Drop-Outs 
Group 1 



Obi actives ; 



798 



Appendix B 



• i 

1 adly stated for the area. 



Assumptions i 

1* Drop-outs can be reduced by the academic process » by educating In 
areas of communications » math and vocational counseling. 



2. Look at critical differences between stay-in and drop-out; Identify I 

participants. f 

(a) Student will be primed as to dress, talk, and promptness to go I 

Into business. I 



(b) Business primed to be gentle and understanding of non-standard 
potential employee. 

(c) Teachers primed to develop their teaching geared to felt needs 
of students In math and communication Skills. 



11th grade student 
regards 12th grade 
Student as leader 



In-Service 



Student visits job 



Student gathers data 
on needs of Job 
expectations 



Student develops 
program from his 
need 








image: 12th grader stayed 

in school 




peer ^o has gone through 
on-Job training will commu- 
nicate to 11th grade novices 
problems to overcome 







3 
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Project Designed from Case Study* 

* 

$ 

Drop-Outs 

Group II , 

Assumptions ; 

1. Set up 6 months lead time for planning 

2. $30^000 budget 

3. Involve parents and children in planning with business people 

4. This is not a complete plan 

ft 

Operations ; 

1* ConMunications lab with media and field experience approach 

2. Listening, speaking, writing, reading about vocations 

3. Special math labs with hardware (CAI, calculators, etc*) 

4. Outpost outside of school 

5. Guaranteed performance contracting 

6* Hands-off experience in a career development approach 

7. Give experience with vocational in and out of school 

8. Bring in outside agencies 

9. Educational program with a person on a job 
10* Work-study approach 

11* Big brother approach from the community 
Evaluation ; 

1. Baseline data - describe population and look for causative factors 

2. Take 100 students at Jefferson feeder schools within ages 14-15 in 
educational program on the job 

3. Ten-year program to phase it in • 

4. 1st year - pilot with 100 children at Jefferson 
Sample includes only high-risk studentr. 

®iao 



5 . 



Evaluation design ; 
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1* Needs assessment from state, community, parents and children 

2. Feedback in planning phase — time-table, critical decisions 

3. 100 children — a cross-section of the student body 

4. Diagnostic information to begin student profiles in skills 

5. Set up feedback system to students and teachers 

6. Feedback from community, business and parents on attitudes, behavior, 
and performance 

% 

7. Periodic status reports to teachers, principals, and others 
8* Note drop-outs in sample 

9. For objectives 3 and 4 use knowledge test, unobtrusive measures 
and open-ended questionnaires 



Communication 

Skills 


*B.O. for objectives (criterion reference) 

Validate B.O. by sampling employee designated outstanding 
in communication. 


Math 


* 


Manpower 


* 


Needs 


. 


Training 


* 


Voc* Ed. Needs 





* Behavioral Objectives 
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Case Study Submitted to Work Groups 
Community Involvement 

I* Short Narrative Description * 

.Inner-city school systems lack qualified* teaching personnel to reduce 

pupil-teacher ratio to levels at which st«adents can effectively benefit from 

classroom instruction. Furthermore, the cost of employing additional certi- 

« 

floated teaching personnel with the present financial crises is prohibitive. 
Therefore, a project to train paraprofessionals , recruited from the inner- 
city neighborhoods, was initiated to assist classroom teachers. 

IX. Outline of Objectives 

A. To provide more individualized instruction for inner-city children 
by reducing pupil-teacher ratio 

• B. To develop a program to train aides and paraprofessionals 
C. To enlist community support for the schools by training para- 
professionals drawn from the school neighborhoods. 

III. Program Description 

Paraprofessional trainees will be selected from the cossaunity to serve 
as teacher aides and library aides. Pre-service and in-service training, 
conducted by consultants from a local university, will complement training 
experiences in the classroom and school library setting. 

Project Designed from Case Study 
Community Involvement 
Group I 

Assumptions » 

1. Teachers feel a need for reduced class size and increased individual 
instruction. 

2. Teachert> see that aides can help to accomplish these goals, and that 
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this approach is not in conflict with the career of the professional 
teacher. 

$ 

Involvement of participants in planning the program will Increase the 
probability of success of the program. 

A planning period will Increase* the chance of success. 

Reduced pupil-aide/ teacher ratio will result in improved Instruction - 
in some way. 

I 

6. Project objectives are specified and publicized so that there is no 
question about what the project will attempt to do. 

7. Objectives will be stated in measurable terms and participants will 
help decide on criteria of success. 

8. Staff will be available to oversee process and product evaluation. 



Planning t 

1* Project director and evaluators will be hired at the start of the 
planning project. 

2. Participants will be involved throughout the planning period (teachers 
community representatives, students, administrators, university and 
school staff, trainers, state and federal consult^ts). 

3. Evaluators will work with participants to specify and publicize 
objectives. 

4. Planning will involve a review of similar projects and Involvement of 
consultants from pertinent projects. 

5. An evaluation of project capability will take place during the planning 
stage. 

6. Participants will be Involved in planning and conducting project 
evaluation. 

7. The administration and funding is committed to a given time period 
for the project and the evaluation. 



Project Designed from Case Study 
Community Involvement 
Group II 

1. Confec with program staff and determine what they hope to do 
a. Ask them to explicitly state what they hope to accomplish 
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b. Ask them to recommend criteria which might Indicate achievement 
o£ these objectives 

c. Evaluation staff would also recommend criteria 

# 

d. Agreement on criteria 

♦ « 

I 

Determine methodo*logy to observe if criteria are being met 

0 

a. Classroom observation, using video tape 

b. Questionnaire to teacher aides and teacher on how they use their 
time 

c* Achievement testing 

d. Opinionnaire of children 

e. Opinionnaires on the training sessions 

f. Observation on the training sessions 

g. Test on knowledge to be acquired 

h. Observation of performance in classroom 

i. Check address lists to see if aides are from school neighborhoods 

j. Attendance of students 

k. Attendance of community at school meetings 

l. Survey of parents and agencies in the community 



f 






Appendix B 



Sea.slon IV 
Force Field Analysis 

Frr^ta Sessions I and II four broad problem areas were chosen to use in 
a force field analysis. 

The participants were divided into eight groups,* two groups per problem 
area. Each group was directed to work Independently for two hours on their 
particular problem. Each group was to identify the restraining forces and 
pushing forces acting upon their problem area. Restraining forces are those 
forces that inhibit solution of the problem. Pushing forces are those forces 
which would solve the problem. 

After they had worked independently, the two groups working on the same 
problem met together. At this time they compared forces which had been 
identified and made their recommendations. 

The four problem areas were: 

1. Training Research Personnel 

2. Decislon<*Halclng 

3. Measuring Unintended Effects 

« 

4. Process Evaluation 
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Dec is ion-*Haking 
Group I 

Restraining Forces t • 

1; Attitude of administration (principals, supervisor, etc.) 

2. Improper timing of results of i-«^'-^aluation (process - product) 

a. Demand 

b. Supply 

c. Level of specificity 

3. Format and language of evaluation reports 

4. Community attitude and awareness 

5. Administrative considerations 
a. Staff size 



I 

P 



I 






U) 
•iw 
X. . 



% 

■M 



6. 



b. Budget 

c. State testing programs 

d. "Big picture" considerations 

Lack of flexibility in recommendations (usually needs delineation 
of multiple alternatives) 



7. Federal and state guidelines 

a. Conflicting 

b. Frequent changes 

8. Status of evaluators and evaluation Information (improper under- 
standing of worth of evaluative information) 

9. Lack of quality and apparent relevance of much evaluative Information 

10. Lack of clearance from intermediate administrative echelons to school 
supervisors 

11. Resistance and hostile attitudes toward evaluation and Innovating 
programs (personnel, role conflicts, power struggles) 

12. Political considerations 



a. Internal and external 



b. Lay board 
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Training Personnel * 

* 

Group I 

1# Personnel Roles Identified: 
a'. Evaluators 
b* Administrators 
c» Community 

d. Institutions (LEA's» universities) 
e< Teachers 

f . Aides 

g. Researchers 

h. C2 and State 

i. Political decision-makers 

Restraining Forces ; 

1. Lack of interested personnel 

a. Need involvement of all personnel 
b* Proper utilization of all personnel 
c* Lack of coordination of training program 
2* Image of Evaluation 

a. Gaining acceptance of evaluation 

b. Too narrowly-trained experts 

c. Institutional restrictions (certification, etc.) 

d. Sources of recruitment too restrictive 

e. Lack of consensus among research and evaluation personnel re: 
theoretical base of Title I 

3. Need to re-orient Congress to other measures of target population 
progress 

4. Inadequate funding for evaluation 

5. Lack of leadership 
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Recommendations i 

1. OE SHOULD ACCEPT A BROADER VARIETY OF INDICES OF CHANGE. OE SHOULD 

CLARIFY ITS OIW POSITION AS A FACILITATING AGENCY ONLY BETWEEN 
CONGRESS AND THE LEA TO ACCOMPLISH THE EVALUATION PROCESS. OE SHOULD 
UPDATE ITS PHILOSOPHICAL STANCE RE THE EDUCATIONAL INTENT AND OBJEC- 
TIVES OF TITLE I. . . 

2. CONGRESS SHOULD PROVIDE FUNDS FOR YEAR-ROUND TRAINING INSTITUTES TO 
TRAIN PARTICIPANTS IN EVALUATION » FOR ALL ROLES TO ACCOMPLISH MORE 
EFFECTIVE ROLE-PERFORMANCE, SUPPORT, AND PROMOTION OF EVALUATIVE 
PROCESS. NEED APPROPRIATE TRAINING TO THE FUNCTION, NOT TO TRADITIONAL 
INSTITUTIONAL REQUIREI^iENTS. 

3. NEED MORE LOCAL, FEDERAL, AND STATE RESOURCES FOR BOLSTERING EVALUA- 
TION PROCESS (ADDITIONAL PERSONNEL). 



Training Personnel 
Group II 

Restraining Forces 

1. Need to develop trained personnel for program research and evaluation: 

a. Central administrative level 

b. Program direction level 

c. Site level 

2. Training content: 

a. Research procedures 

1. Examination of objectives 

2. Selection of appropriate methods 

3. Communication of problems information-needs 

b. Data-processing and analysis 

1. Interviews 

2. Questionnaires 

3. Observational 

4. Standardized or special tests 

5. Other 

6. Evaluation in terms useful for decision-making 
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c. Personnel Qualifications 

1. college background 

2. school background 
3* coimnunity 



Recomaendattona t 

1. SPONSORSHIP BY OE OF SCHOLARSHIPS AND GRANTS FOR UNDERGRADUATE AND 
GRADUATE MAJORS IN R&D TECHNIQUES 

2. REQUIREMENT THAT RESEARCH PERSONNEL AND SCHOOL PERSONNEL AT ALL LEVELS 
PARTICIPATE IN TRAINING SESSIONS SUCH AS WORKSHOPS » IN-SERVICE TRAIN- 
ING» AND INSTITUTES: 

a. TO GET RESEARCH PERSONNEL TO UNDERSTAND SCHOOL PROBLEMS 

b. TO BE SURE THAT SCHOOL PERSONNEL UNDERSTAND THE PROBLEMS OF RE- 
SEARCH SPECIFICALLY APPLICABLE TO THE PROGRAM OR PROJECT INVOLVED 

3. FACILITATE THE EXCHANGE AND INFORMATION AMONG VARIOUS LEVELS OF PRO- 
GRAM PERSONNEL PARTICIPAITTS 

4. OE SHOULD ACCEPT A BROADER RANGE OF INDICES OF CHANGE 
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Measuring Unanticipated Effects 
Group 1 



Restraining Forces ; 

. I. Inadequate planning time 

2. Lack of sufficient knowledge to '.anticipate the unanticipated 

3, Lack of, or inadequate communication, among involved personnel 

4* Inadequate supply of funds for additional staff and staff train- 
ing 

5. Inadequate commitment to compensatory funding 

6, Community reaction: inadequate Information to “predict” reactions 

the general state of relations between community and schools 

7, Inadequacy or fnappropriateness of measurement techniques 
a. Too rigid adherence to original evaluation design 

bi> Too narrow or too discrete objectives 

8. Insecurity in reporting results other than standardized tests — 
restricted view of outcomes 

9* Changes in personnel functions and roles . 

10. Requirements, and changes in requirements, of state departments 

11. Inadequate data regarding needs and characteristics of pupils to 
be served 

12. Inadequate communication and dissemination of outcomes (reduces 
chances of learning from experiences of others) 



Pushing Forces : 

1. Pressure for valid account£d>ility 

•Federal 
.State 
. Local 
school 
parents 
community 
pupils * 

2. Deadlines for proposals 
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Recommendations : 



1. PERSONNEL — MORE MONEY. TO ADD AND TRAIN PERSONNEL WITH THE 
SPECIFICATION OF A REQUIRED PERCENTAGE OF FUNDS TO BE .ALLOCATED 
TO EVALUATION FOR USE IN OBTAINING ADDITIONAL SPACE. VIA RENT. 
LEASE, ETC. 

* 

2. PLANNING TIME — TO EXPEDITE TRANSMISSION OF GUIDELINES THROUGH 
CHANNELS (FEDERAL > STATE, AND LOCAL): TO SIMPLIFY GUIDELINES AND 
PERMIT MORE FLEXIBILITY IN APPLICATION BY lea and sea: AND TO 
PROVIDE PLANNING FUNDS. 

3. ASSESSMENT TECHNIQUES — LIMIT EMPHASIS ON STANDARDIZED TEST DATA; 
MORE EMPHASIS SHOULD BE PLACED ON PROCESS . AFFECTIVE MEASURES, AND 
DESCRIPTIVE STUDIES. 



Measuring Unanticipated Effects 
Group II 



Restraining Factors : 

1. Pre/post testing; no process control 

2. Planning -- nonparticipation of program participants, lack of 
communicat ion 

3. Staff resistance to innovation 

4. Rigid guidelines by OL 

5. Unrealistic expectations 

a. Success of project 

b. Interpretation of objectives 

c. Funding 



Pushing Factors : 

1. Continuous seif-correcting evaluation 



Recommendations : 



1. OE SHOULD PROVIDE FUNDS FOR TEACHER TRAINING FOR URBAN TARGET 
AREAS. 

2. SO THAT UNANTICIPATED CONSEQUENCES DO NOT BECOME NEGATIVE EFFECTS 
PARENTS, STUDENTS, AiW STAFF SHOULD BE INVOLVED IN PLANNING AND * 
EVALUATION. 
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3. ENCOURAGE CONTINUOUS SELF-CORRECTING EVALUATION MODEL (SWRL) 

SOUTH WEST REGIONAL LAB. 

4. TECHNICAL ASSISTANCE SHOULD BE PROVIDED TO 

a. DEVELOP A VARIETY OF INSTRUMENTS AND METHODOLOGY 

* . 

b. DEVELOU TECHNICAL SKILLS OF EVALUATIONS AND EVALUATORS. 

a 

5. PROVIDE ADVANCE FUNDING TO PERMIT LEAD TIME FOR PLANNING AND DEVEL* 
OPMENT. 

ft 

6. EXPECTATIONS FOR RESULTS SHOULD BE DETERMINED BY LOCAL PROGRAM PAR- 
TICIPANTS IN TERMS OF WHAT THEY CONSIDER REALISTIC GOALS FOR TIME 
PERIODS. 

7. IF UNANTICIPATED CONSEQUENCES ARE SOUGHT BY OE, THEN A DEFINITE 
STATEMENT IN THE GUIDELINES SHOULD SAY THAT STAFF BE GEARED TO BE- 
COMING AWARE OF, DETERMINING WHICH ARE, AND REPORTING UNANTICIPATED 
CONSEQUENCES. 

8. OE NEEDS TO BE AWARE THAT THERE EXISTS STAFF RESISTANCE TO CHANGE 
WHICH MANIFESTS ITSELF IN NUMEROUS UNANTICIPATED CONSEQUENCES. 



Process Evaluation 
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Group I 

Restraining Forces : 

1. Time Consuming: 

a. Instruction 

b. Preplanning time 

c. Personnel 

1. preplanning 

2. operational 

d. Training 

2. Traditional emphasis on product evaluation 

3. Funds 

4. Fear of evaluation of staff members 

5. Erosion of sanctions 

6. Guidelines 

7. iSea's rural orientation 

8. Lack of communication between OE and sea 

9. Organization of school systems 

a. Traditionally based on line and staff 



Pushing Forces ; 

1. Moving ax^ay from ” crash” programs 

2. Teacher demand for involvement in preplanning and operation 

3. Community involvement in planning and evaluation 

a. Students 

b. Parents 

c. Industry 

d. Agencies 

4. Groining recognition of the inadequacies of traditional evaluation 

m 
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5. Growing body of literature 

6. Accountability of funds 

7. In-'service education ' 

, 8. Awareness that programs do not follow submitted plans 

9* Use of incentives-financial) etf:. for students and staff 

10. Clear guidelines 

a. Federal 

b. State 

» 

c. Bocal 

11. Erosion of sanctions 
Recommendations ; 

I. COMMITMENT OF A PERCENTAGE OF FUNDS TO “PROCESS EVALUATION” 

A. FUNDS TO BE ALLOCATED FOR: 

1. TEACHER PLANNING TI>1£, IN THE SUMMER AND DURX?4G THE YEAR 

2. COMMUNITY PLANNING, IN THE SUMMER AND DURING THE YEAR 

3. PROJECT STAFF 

II. NEW EVALUATION ROLE 

A. “ON-SITE” EVALUATION AS OPPOSED TO CENTRAL STAFF 

B. NEED OF EVALUATOR TO TRAIN PARTICIPANTS IN “PROCESS EVALUA- 
TION." 

Process Evaluation 
Group II 

Restraining Forces ; 

1. Process evaluation requires an unmanageable number of methodologies 

2. Inadequate communication between operations and evaluation staff 

3. Inadequate resource allocation by staff (materials, equipment, funds 
time) 

. Vagueness of most pro ram plans 
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5. Activity level of instructfonal staff; other staff apathy 

6. Record-keeping overload 

7. Humanism of staff 

8. Question of policy or negotiated working conditions for staff 
9* Lack of specificity in guidelines 

10. Threatening element to teachers, administrators 

11. Undesirability of additional adults in classrooms 

Pushing Forces : 

1. Offers direct benefit to operation staff, engendering their 
support 

2. Provides immediate feedback 

3. Change in project objectives enhances desirability of process 
evaluation 

4. Permits the emergence of new, unanticipated variables 

5. Permits replication, through ability to define a plan 

6. Has potential for individualization of instruction 

7. Meets administrative information needs, improving accountability 

8. Permits or aids cost /benefit analysis study 

9. Community interest and group pressure for information places a 
priority on immediacy of information 

10. High numbers of evaluation staff per project would be reduced 

11. Develops an interested, committed, and demanding staff in the 
process 

12. Offers availability of differentiated staff for locals 

13. Focuses attention on evaluation, producing the halo or Hawthorne 
syndrome 

14. Meets a basic need of all, particularly the disadvantaged for 
Instant feedback on programs 
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Recommendations : 



1. PROVIDE GUIDELINES, DEVELOP A SAMPLE OF STANDARD MATERIALS; 
PROVIDE IN-SERVICE TRAINING 

# 

2. BUILD STRATEGY AND RESOURCES INTO THE FUNDING PROPOSAL, E.G., 

PERFORMANCE CONTRACT - . 

3. PROVIDE ADE'^UATE BUDGET AND RESOURCES TO PROVIDE FOR PROJECT 
REVISION AND RESULTANT CHANGES IN EXPECTATIONS 

4. REQUIRE MORE PROGRAM SPECIFICITY IN TERMS OF PROCESS EVALUATION 
BEFORE FUNDING (I.E. , THE BUDGET SHOULD REFLECT A COMMITMENT TO 
EVALUATION/MANAGEMENT/OPERATION PLANS) . 

5. GRADUATED INCENTIVES FOR STAFF TRAINING 

« 

6. PROVIDE AIDES FOR RECORD-KEEPING 

% 

7. LONG-RANGE PROFESSIONAL TRAINING * ’ 

8. NATIONALLY, OE SHOULD FUND TRAINING IN PROCESS EVALUATION AT ALL 
LEVELS 



9 . 



CONSORTIA OF L.E.A., S.E.A., USOE SHOULD JOINTLY DEVELOP GUIDELINES 



